# Respond to an incident

> A process map exported from forks.world (https://forks.world). Author: Brian. Source: https://forks.world/p/cmv0iedbx004kjw060iqtrof5
> It describes who does what, in what order, and with which technology. Read the plain-language sections first; the JSON at the end is the exact, complete data.

## Summary

- 18 steps across 2 layers
- Persona: Incident commander, On-call engineer, Communications, Customers
- Technology / systems: Monitoring

## How to read this file

- A process is made of **layers**. The main layer is the top level; any step of kind `subprocess` opens a detail layer with its own lanes and steps.
- **Lanes** are either persona (people/roles) or technology (systems). Each step belongs to one lane; `using` lists tools a person uses for that step.
- Step kinds: `start`, `task`, `decision` (its arrows carry the answer as a label), `subprocess`, `end`.

## Respond to an incident

Lanes:
- Monitoring: technology / system (id `mon`)
- Incident commander: person / role (id `ic`)
- On-call engineer: person / role (id `eng`)
- Communications: person / role (id `comms`)
- Customers: person / role (id `cust`)

Steps, in the order they run:

1. **An alert fires: error rate spikes** [start; Monitoring; id `i1`]
   - then Acknowledge within 5 minutes
2. **Acknowledge within 5 minutes** [task; On-call engineer; id `i2`]
   - then Real incident or false alarm?
3. **Real incident or false alarm?** [decision; On-call engineer; id `i3`]
   - if "Real" go to Declare an incident and assign roles; if "False alarm" go to Tune the alert and close it
4. **Declare an incident and assign roles** [task; Incident commander; id `i4`]
   - then Mitigate first: roll back or move traffic
5. **Mitigate first: roll back or move traffic** [task; On-call engineer; id `i5`]
   - then Post a status update
6. **Post a status update** [task; Communications; id `i6`]
   - then Is service restored?
7. **Is service restored?** [decision; On-call engineer; id `i7`]
   - if "Yes" go to Announce: resolved; if "No" go to Bring in more experts; update every 30 minutes
8. **Announce: resolved** [task; Communications; id `i8`]
   - then Schedule the post-mortem within 3 days
9. **Schedule the post-mortem within 3 days** [subprocess; Incident commander; id `i9`], opens the detail layer "The blameless post-mortem"
   - (no next step recorded)
10. **Bring in more experts; update every 30 minutes** [task; Incident commander; id `i7a`]
   - then Mitigate first: roll back or move traffic
11. **Tune the alert and close it** [end; On-call engineer; id `i3a`]

```mermaid
flowchart LR
  subgraph l_mon["Monitoring"]
    n_i1(["An alert fires: error rate spikes"])
  end
  subgraph l_ic["Incident commander"]
    n_i4["Declare an incident and assign roles"]
    n_i7a["Bring in more experts; update every 30 minutes"]
    n_i9[["Schedule the post-mortem within 3 days"]]
  end
  subgraph l_eng["On-call engineer"]
    n_i2["Acknowledge within 5 minutes"]
    n_i3{"Real incident or false alarm?"}
    n_i3a(["Tune the alert and close it"])
    n_i5["Mitigate first: roll back or move traffic"]
    n_i7{"Is service restored?"}
  end
  subgraph l_comms["Communications"]
    n_i6["Post a status update"]
    n_i8["Announce: resolved"]
  end
  n_i1 --> n_i2
  n_i2 --> n_i3
  n_i3 -->|"Real"| n_i4
  n_i3 -->|"False alarm"| n_i3a
  n_i4 --> n_i5
  n_i5 --> n_i6
  n_i6 --> n_i7
  n_i7 -->|"Yes"| n_i8
  n_i7 -->|"No"| n_i7a
  n_i7a --> n_i5
  n_i8 --> n_i9
```

### The blameless post-mortem (detail of step "Schedule the post-mortem within 3 days")

Lanes:
- Incident commander: person / role (id `ic`)

Steps, in the order they run:

1. **Write the timeline of what happened** [start; Incident commander; id `p1`]
   - then Ask why, five times, to find the root cause
2. **Ask why, five times, to find the root cause** [task; Incident commander; id `p2`]
   - then Was it an individual's mistake or a system gap?
3. **Was it an individual's mistake or a system gap?** [decision; Incident commander; id `p3`]
   - if "A system gap" go to Fix the system: tests, alerts, guardrails; if "An individual's mistake" go to Ask what made the mistake easy to make
4. **Fix the system: tests, alerts, guardrails** [task; Incident commander; id `p4`]
   - then Action items with owners and dates
5. **Action items with owners and dates** [task; Incident commander; id `p6`]
   - then Publish the post-mortem to the whole company
6. **Publish the post-mortem to the whole company** [end; Incident commander; id `p7`]
7. **Ask what made the mistake easy to make** [task; Incident commander; id `p5`]
   - then Fix the system: tests, alerts, guardrails

```mermaid
flowchart LR
  subgraph l_ic["Incident commander"]
    n_p1(["Write the timeline of what happened"])
    n_p2["Ask why, five times, to find the root cause"]
    n_p3{"Was it an individual's mistake or a system gap?"}
    n_p5["Ask what made the mistake easy to make"]
    n_p4["Fix the system: tests, alerts, guardrails"]
    n_p6["Action items with owners and dates"]
    n_p7(["Publish the post-mortem to the whole company"])
  end
  n_p1 --> n_p2
  n_p2 --> n_p3
  n_p3 -->|"A system gap"| n_p4
  n_p3 -->|"An individual's mistake"| n_p5
  n_p5 --> n_p4
  n_p4 --> n_p6
  n_p6 --> n_p7
```

## Full data (JSON)

```json
{
  "title": "Respond to an incident",
  "maps": {
    "m_pm": {
      "id": "m_pm",
      "lanes": [
        {
          "id": "ic",
          "name": "Incident commander",
          "type": "person"
        }
      ],
      "steps": [
        {
          "id": "p1",
          "kind": "start",
          "lane": "ic",
          "next": [
            {
              "to": "p2"
            }
          ],
          "uses": [],
          "label": "Write the timeline of what happened"
        },
        {
          "id": "p2",
          "kind": "task",
          "lane": "ic",
          "next": [
            {
              "to": "p3"
            }
          ],
          "uses": [],
          "label": "Ask why, five times, to find the root cause"
        },
        {
          "id": "p3",
          "kind": "decision",
          "lane": "ic",
          "next": [
            {
              "to": "p4",
              "label": "A system gap"
            },
            {
              "to": "p5",
              "label": "An individual's mistake"
            }
          ],
          "uses": [],
          "label": "Was it an individual's mistake or a system gap?"
        },
        {
          "id": "p5",
          "kind": "task",
          "lane": "ic",
          "next": [
            {
              "to": "p4"
            }
          ],
          "uses": [],
          "label": "Ask what made the mistake easy to make"
        },
        {
          "id": "p4",
          "kind": "task",
          "lane": "ic",
          "next": [
            {
              "to": "p6"
            }
          ],
          "uses": [],
          "label": "Fix the system: tests, alerts, guardrails"
        },
        {
          "id": "p6",
          "kind": "task",
          "lane": "ic",
          "next": [
            {
              "to": "p7"
            }
          ],
          "uses": [],
          "label": "Action items with owners and dates"
        },
        {
          "id": "p7",
          "kind": "end",
          "lane": "ic",
          "next": [],
          "uses": [],
          "label": "Publish the post-mortem to the whole company"
        }
      ],
      "title": "The blameless post-mortem",
      "parent": {
        "map": "m_root",
        "step": "i9"
      }
    },
    "m_root": {
      "id": "m_root",
      "lanes": [
        {
          "id": "mon",
          "name": "Monitoring",
          "type": "system"
        },
        {
          "id": "ic",
          "name": "Incident commander",
          "type": "person"
        },
        {
          "id": "eng",
          "name": "On-call engineer",
          "type": "person"
        },
        {
          "id": "comms",
          "name": "Communications",
          "type": "person"
        },
        {
          "id": "cust",
          "name": "Customers",
          "type": "person"
        }
      ],
      "steps": [
        {
          "id": "i1",
          "kind": "start",
          "lane": "mon",
          "next": [
            {
              "to": "i2"
            }
          ],
          "uses": [],
          "label": "An alert fires: error rate spikes"
        },
        {
          "id": "i2",
          "kind": "task",
          "lane": "eng",
          "next": [
            {
              "to": "i3"
            }
          ],
          "uses": [],
          "label": "Acknowledge within 5 minutes"
        },
        {
          "id": "i3",
          "kind": "decision",
          "lane": "eng",
          "next": [
            {
              "to": "i4",
              "label": "Real"
            },
            {
              "to": "i3a",
              "label": "False alarm"
            }
          ],
          "uses": [],
          "label": "Real incident or false alarm?"
        },
        {
          "id": "i3a",
          "kind": "end",
          "lane": "eng",
          "next": [],
          "uses": [],
          "label": "Tune the alert and close it"
        },
        {
          "id": "i4",
          "kind": "task",
          "lane": "ic",
          "next": [
            {
              "to": "i5"
            }
          ],
          "uses": [],
          "label": "Declare an incident and assign roles"
        },
        {
          "id": "i5",
          "kind": "task",
          "lane": "eng",
          "next": [
            {
              "to": "i6"
            }
          ],
          "uses": [],
          "label": "Mitigate first: roll back or move traffic"
        },
        {
          "id": "i6",
          "kind": "task",
          "lane": "comms",
          "next": [
            {
              "to": "i7"
            }
          ],
          "uses": [],
          "label": "Post a status update"
        },
        {
          "id": "i7",
          "kind": "decision",
          "lane": "eng",
          "next": [
            {
              "to": "i8",
              "label": "Yes"
            },
            {
              "to": "i7a",
              "label": "No"
            }
          ],
          "uses": [],
          "label": "Is service restored?"
        },
        {
          "id": "i7a",
          "kind": "task",
          "lane": "ic",
          "next": [
            {
              "to": "i5"
            }
          ],
          "uses": [],
          "label": "Bring in more experts; update every 30 minutes"
        },
        {
          "id": "i8",
          "kind": "task",
          "lane": "comms",
          "next": [
            {
              "to": "i9"
            }
          ],
          "uses": [],
          "label": "Announce: resolved"
        },
        {
          "id": "i9",
          "kind": "subprocess",
          "lane": "ic",
          "next": [],
          "uses": [],
          "child": "m_pm",
          "label": "Schedule the post-mortem within 3 days"
        }
      ],
      "title": "Respond to an incident",
      "parent": null
    }
  }
}
```