Runbook: service is down
Symptoms#
- Requests return server errors, or time out, on a path that normally works.
- The error rate on one route rises sharply while traffic stays flat.
- Several reports of the same failure arrive within a few minutes.
One failing request from one person is not this runbook. Confirm the problem is broader before you escalate a whole team out of their afternoon.
Checks#
Load the failing address yourself, from a network you do not control. A local problem and an outage look identical from your own laptop.
List deploys from the last few hours. A change landing minutes before the symptom is the strongest lead you will get.
Database, queue, cache, and any third party in the failing path. Open their status pages before you read your own logs — see Dependency outage.
Take one real failing request and follow its error and stack trace. Aggregates tell you how much; a single trace tells you what.
Connection pools, disk, rate limits and quotas all fail in ways that look like ordinary bugs. Database degraded covers the common shapes.
Actions#
- If a recent change is implicated, roll it back. Do not wait until you understand it.
- If a dependency is down, degrade deliberately: serve cached content, queue writes, or disable the affected feature behind a flag.
- If capacity is the cause, restore headroom now and size it properly later.
- Record each action with its timestamp as you take it, not afterwards.
Two people taking different mitigating actions at once produces a system neither of them can reason about. Say what you are about to do in the channel before you do it, even when it is obvious.
Escalate when#
Service is not restored within your team's agreed window, the fix would touch stored data, or customer data may have been exposed. See Escalation.