Runbook: database degraded
Symptoms#
- Latency rises on database-backed routes while static paths stay fast.
- Connection errors or pool-exhaustion messages in the application logs.
- Replication lag growing rather than recovering.
- CPU, memory or disk on the instance near its ceiling.
Checks#
What is running right now, and for how long. One long transaction holding a lock explains more slowdowns than any other single cause.
Pool exhaustion looks like an application bug from the application's side. Compare current connections with the configured maximum before you go looking for one.
A full disk fails writes in ways that read as unrelated errors. Cache hit rate falling off a cliff means the working set no longer fits.
A new query without an index, an N+1 introduced in a refactor, or a migration still running are the usual recent causes.
If reads are served from a replica, a lagging replica returns stale data silently — which is worse than an error, because nothing alerts.
Actions#
- Kill the blocking query, not every query. Note what it was before you kill it; that note is the fix.
- Roll back a recent change if the timing implicates one — see Rollback.
- Shed load deliberately: disable a heavy background job, reduce a batch size, rate-limit an expensive endpoint.
- Add headroom if capacity genuinely is the cause, and size it properly afterwards rather than during.
A restart discards the evidence, takes the cache with it, and on a busy system produces a thundering herd on the way back up. Restarting is a decision with a reason, not a first move.
Fill this in: the real commands for your database, and the numbers that mean trouble for your instance size.