Runbook: rollback
Rolling back is the default response, not the last resort. A revert you later decide was unnecessary costs one deploy. An outage debugged live costs everyone.
Before you roll back#
- Identify the last release that was known good, by commit, not by "yesterday".
- Check whether the suspect change included a database migration. If it did, read the data section below before touching anything.
- Tell the incident channel what you are about to do, in one line.
Rolling back#
Through the normal pipeline, rather than hand-editing anything in production. A hand-edit is a change nobody can find later.
A queued or blocked deploy looks exactly like a completed one from the terminal you started it in.
Repeat the request that was failing. Do not rely on the graph flattening — it lags, and it recovers for other reasons too.
Read the version or commit back from the running service, not from your own notes.
When data is involved#
Reverting code may leave the schema ahead of it, which is usually harmless. Reverting the schema while newer code still expects it usually is not. Anything that deleted or overwrote rows needs a restore plan and a second person, not a quick decision under pressure.
After the rollback#
- The original problem still exists. You have only stopped it reaching users.
- Open the fix as a normal change, with a test that would have caught it.
- Leave a note on the reverted pull request explaining why, so nobody re-lands it unchanged next week.