Runbook: dependency outage
Confirm it is them#
- Open their status page, and check it against their own API rather than trusting the page alone. Status pages update late.
- Call the dependency directly from a machine you control, outside your application.
- Check whether the failure is total or partial: one region, one endpoint, one account, or a raised error rate.
A partial outage is the dangerous shape, because retries succeed often enough to hide it and slow enough to exhaust your own capacity.
Degrade deliberately#
Reads you can serve from cache
Serve stale, and say it is stale where the user can see. A slightly old answer beats an error for almost every read.
Writes you can queue
Accept the request, persist the intent, and replay when the provider returns. Tell the user it is pending rather than pretending it completed.
Features you can switch off
A flag that removes the dependent feature keeps the rest of the product usable. Decide the flag now; discovering you do not have one is a finding for the write-up.
Work you cannot do at all
Fail fast and say so. A ten-second timeout on every request takes your own service down alongside theirs.
Retry with a cap, exponential backoff and jitter, or not at all. An unbounded retry loop turns their outage into your outage, and delays their recovery once they are back.
Tell people#
Users forgive an outage they were told about far more readily than a silent one. Say what is not working, what still works, and when you will next say something — then say something then, even with no news.
Recovering#
Bring traffic back gradually. Everything you queued will want to run at once, and a stampede on a service that has just recovered puts it straight back down.