Alerts and paging
The test for a new alert#
Before adding one, answer: what would the person woken by this do in the first five minutes? If the answer is "look at it", the alert is a graph. If the answer is "follow this runbook", link the runbook from the alert itself.
What is allowed to page#
Fill this in: the thresholds below are shapes, not values. Replace them with the numbers your service actually uses, and link each one to the runbook it expects.
| Signal | Pages when | Runbook |
|---|---|---|
| Availability | Core flow failing for everybody | Service is down |
| Error rate | One route breaks its threshold for several minutes | Service is down |
| Latency | The path a user waits on crosses its budget | Database degraded |
| Saturation | Connections, disk or queue depth near a hard limit | Database degraded |
| Dependency | A provider you cannot route around is failing | Dependency outage |
Everything else goes to a channel, a ticket, or nowhere.
Tuning a noisy alert#
An alert that fired and needed nothing gets dealt with the same day, not "when there is time". Three options, in order of preference: raise the threshold to where action actually starts, widen the window so a blip does not qualify, or delete it because the thing it watches is no longer worth watching.
Record what you changed and why on the alert itself. The next person to see it firing will otherwise re-tighten it from first principles.
A rotation that gets three false pages a week stops reading the fourth one carefully. Every unactionable alert spends the attention you are saving for the real one.
Alerts nobody owns#
Every alert has a team's name on it. An alert with no owner is deleted after a sensible notice period — not kept in case it matters, because "in case it matters" is exactly what makes a rotation unsurvivable.