Runbooks and on-call
Written for the worst ten minutes of someone's week. Short, ordered, imperative — because it gets read by a person who is stressed and quite possibly on a phone.
Stop the bleeding before you diagnose, say out loud that you are on it, and write as you go. Timestamps and commands are impossible to reconstruct afterwards, and they are most of what the write-up needs.
Runbooks#
Writing a new runbook#
Name a symptom in the title, not a system. People search for what they can see — "checkout returns an error" — not for the component that turns out to be responsible. Then follow the same shape every time: symptoms, checks, actions, escalation.
Keep every step executable. A step that says "investigate the queue" is a wish. A step that gives the command and the number that means trouble is a runbook.
Keeping them true#
Runbooks rot faster than any other documentation, because the systems they describe are exactly the ones changing. The cheapest fix is a habit: whoever used a runbook during an incident edits it straight afterwards, while the mismatch is still annoying.