Update: Issue is resolved! Database is back online.

Retrospective

We had escalating database issues starting from last Friday, July 14th, leading up to a two-hour outage on Monday, July 17 from 10:50am-1pm PT.

This is the worst outage Manifold Markets has ever had, and it is not up to our standards. We can and will do better.

After debugging the issues extensively and a helpful call with the Supabase Team, we now have a good grasp of all the issues.

The two hour outage was caused by a reboot loop when we manually restarted our database in an attempt to fix other issues.

Read what Supabase wrote as a retrospective on this issue.
The larger problem that was slowing down our database was with a table user_contract_metrics that was big and not being auto-vacuumed leading to high disk usage.

We’ve since resolved all issues, and re-enabled most site functionality that was disabled while we were debugging the problem.

~~The only remaining task is to re-enable the API.~~ API is back too!

We’re taking many steps to make sure this doesn’t happen again or has lesser impact if it does.

Weekly cleaning of database to fix bloated tables
Migrate off of data json columns which use a GIN index that multiplies this issue
When database CPU threshold reaches 40%, assign a high priority task to find and fix issues
Upgrade to a Supabase plan with priority support so that database doesn’t go down for as long.

Aside from the technical and operation improvements, we want to communicate better with users.