Respond to an incident
Process map by Brian. 18 steps, 2 layers. Tags: business, engineering.
- Monitoring: An alert fires: error rate spikes
- On-call engineer: Acknowledge within 5 minutes
- On-call engineer: Real incident or false alarm?
- Incident commander: Declare an incident and assign roles
- On-call engineer: Mitigate first: roll back or move traffic
- Communications: Post a status update
- On-call engineer: Is service restored?
- Communications: Announce: resolved
- Incident commander: Schedule the post-mortem within 3 days
- Incident commander: Bring in more experts; update every 30 minutes
- On-call engineer: Tune the alert and close it