# Ship a product release without a 2 a.m. page

> A process map exported from forks.world (https://forks.world). Author: Brian. Source: https://forks.world/p/cmv0iecvj004fjw06uy27axj9
> It describes who does what, in what order, and with which technology. Read the plain-language sections first; the JSON at the end is the exact, complete data.

## Summary

- 23 steps across 2 layers
- Persona: Product manager, Engineers, QA, Operations, Customers
- Technology / systems: CI/CD pipeline

## Pain points

Where this process hurts, worst first:

- **Roll back and alert the team** (Ship a product release without a 2 a.m. page; critical): A rollback at midnight, after a launch nobody could stop
- **Fix the failing tests** (Ship a product release without a 2 a.m. page; painful): Flaky tests: failures that are not real bugs

## How to read this file

- A process is made of **layers**. The main layer is the top level; any step of kind `subprocess` opens a detail layer with its own lanes and steps.
- **Lanes** are either persona (people/roles) or technology (systems). Each step belongs to one lane; `using` lists tools a person uses for that step.
- Step kinds: `start`, `task`, `decision` (its arrows carry the answer as a label), `subprocess`, `end`.

## Ship a product release without a 2 a.m. page

Lanes:
- Product manager: person / role (id `pm`)
- Engineers: person / role (id `dev`)
- QA: person / role (id `qa`)
- CI/CD pipeline: technology / system (id `ci`)
- Operations: person / role (id `ops`)
- Customers: person / role (id `cust`)

Steps, in the order they run:

1. **Release scope frozen** [start; Product manager; id `r1`]
   - then Features merged into the release branch
2. **Features merged into the release branch** [task; Engineers; id `r2`]
   - then Automated tests run
3. **Automated tests run** [task; CI/CD pipeline; id `r3`]
   - then All green?
4. **All green?** [decision; CI/CD pipeline; id `r4`]
   - if "Yes" go to Manual test pass and exploratory testing; if "No" go to Fix the failing tests
5. **Manual test pass and exploratory testing** [task; QA; id `r5`]
   - then Any blocking bugs?
6. **Any blocking bugs?** [decision; QA; id `r6`]
   - if "Clear to ship" go to Go / no-go meeting; if "Blockers found" go to Fix and re-verify
7. **Go / no-go meeting** [task; Product manager; id `r7`]
   - then Go?
8. **Go?** [decision; Product manager; id `r8`]
   - if "Go" go to Deploy to 5% of users first; if "No go" go to Choose a new date and tell everyone
9. **Deploy to 5% of users first** [subprocess; CI/CD pipeline; id `r9`], opens the detail layer "Canary deploy"
   - then Metrics healthy?
10. **Metrics healthy?** [decision; Operations; id `r10`]
   - if "Yes" go to Roll out to everyone; if "No" go to Roll back and alert the team
11. **Roll out to everyone** [task; CI/CD pipeline; id `r11`]
   - then Announce: release notes and email
12. **Announce: release notes and email** [task; Product manager; id `r12`]
   - then Customers see the new feature
13. **Customers see the new feature** [end; Customers; id `r13`]
14. **Roll back and alert the team** [task; Operations; id `r10a`] ⚠ PAIN POINT (critical): A rollback at midnight, after a launch nobody could stop
   - then Features merged into the release branch
15. **Choose a new date and tell everyone** [task; Product manager; id `r8a`]
   - then Features merged into the release branch
16. **Fix and re-verify** [task; Engineers; id `r6a`]
   - then Automated tests run
17. **Fix the failing tests** [task; Engineers; id `r3a`] ⚠ PAIN POINT (painful): Flaky tests: failures that are not real bugs
   - then Automated tests run

```mermaid
flowchart LR
  subgraph l_pm["Product manager"]
    n_r1(["Release scope frozen"])
    n_r7["Go / no-go meeting"]
    n_r8{"Go?"}
    n_r8a["Choose a new date and tell everyone"]
    n_r12["Announce: release notes and email"]
  end
  subgraph l_dev["Engineers"]
    n_r2["Features merged into the release branch"]
    n_r3a["Fix the failing tests"]
    n_r6a["Fix and re-verify"]
  end
  subgraph l_qa["QA"]
    n_r5["Manual test pass and exploratory testing"]
    n_r6{"Any blocking bugs?"}
  end
  subgraph l_ci["CI/CD pipeline"]
    n_r3["Automated tests run"]
    n_r4{"All green?"}
    n_r9[["Deploy to 5% of users first"]]
    n_r11["Roll out to everyone"]
  end
  subgraph l_ops["Operations"]
    n_r10{"Metrics healthy?"}
    n_r10a["Roll back and alert the team"]
  end
  subgraph l_cust["Customers"]
    n_r13(["Customers see the new feature"])
  end
  n_r1 --> n_r2
  n_r2 --> n_r3
  n_r3 --> n_r4
  n_r4 -->|"Yes"| n_r5
  n_r4 -->|"No"| n_r3a
  n_r3a --> n_r3
  n_r5 --> n_r6
  n_r6 -->|"Clear to ship"| n_r7
  n_r6 -->|"Blockers found"| n_r6a
  n_r6a --> n_r3
  n_r7 --> n_r8
  n_r8 -->|"Go"| n_r9
  n_r8 -->|"No go"| n_r8a
  n_r8a --> n_r2
  n_r9 --> n_r10
  n_r10 -->|"Yes"| n_r11
  n_r10 -->|"No"| n_r10a
  n_r10a --> n_r2
  n_r11 --> n_r12
  n_r12 --> n_r13
```

### Canary deploy (detail of step "Deploy to 5% of users first")

Lanes:
- CI/CD pipeline: technology / system (id `ci`)
- Operations: person / role (id `ops`)

Steps, in the order they run:

1. **Route 5% of traffic to the new version** [start; CI/CD pipeline; id `c1`]
   - then Watch errors, speed and sign-ups for an hour
2. **Watch errors, speed and sign-ups for an hour** [task; Operations; id `c2`]
   - then Worse than the old version?
3. **Worse than the old version?** [decision; Operations; id `c3`]
   - if "Yes" go to Automatic rollback; if "No" go to Increase to 25%, then 50%
4. **Automatic rollback** [task; CI/CD pipeline; id `c4`]
   - then Decision logged
5. **Decision logged** [end; Operations; id `c6`]
6. **Increase to 25%, then 50%** [task; CI/CD pipeline; id `c5`]
   - then Decision logged

```mermaid
flowchart LR
  subgraph l_ci["CI/CD pipeline"]
    n_c1(["Route 5% of traffic to the new version"])
    n_c4["Automatic rollback"]
    n_c5["Increase to 25%, then 50%"]
  end
  subgraph l_ops["Operations"]
    n_c2["Watch errors, speed and sign-ups for an hour"]
    n_c3{"Worse than the old version?"}
    n_c6(["Decision logged"])
  end
  n_c1 --> n_c2
  n_c2 --> n_c3
  n_c3 -->|"Yes"| n_c4
  n_c3 -->|"No"| n_c5
  n_c4 --> n_c6
  n_c5 --> n_c6
```

## Full data (JSON)

```json
{
  "title": "Ship a product release without a 2 a.m. page",
  "maps": {
    "m_root": {
      "id": "m_root",
      "lanes": [
        {
          "id": "pm",
          "name": "Product manager",
          "type": "person"
        },
        {
          "id": "dev",
          "name": "Engineers",
          "type": "person"
        },
        {
          "id": "qa",
          "name": "QA",
          "type": "person"
        },
        {
          "id": "ci",
          "name": "CI/CD pipeline",
          "type": "system"
        },
        {
          "id": "ops",
          "name": "Operations",
          "type": "person"
        },
        {
          "id": "cust",
          "name": "Customers",
          "type": "person"
        }
      ],
      "steps": [
        {
          "id": "r1",
          "kind": "start",
          "lane": "pm",
          "next": [
            {
              "to": "r2"
            }
          ],
          "uses": [],
          "label": "Release scope frozen"
        },
        {
          "id": "r2",
          "kind": "task",
          "lane": "dev",
          "next": [
            {
              "to": "r3"
            }
          ],
          "uses": [],
          "label": "Features merged into the release branch"
        },
        {
          "id": "r3",
          "kind": "task",
          "lane": "ci",
          "next": [
            {
              "to": "r4"
            }
          ],
          "uses": [],
          "label": "Automated tests run"
        },
        {
          "id": "r4",
          "kind": "decision",
          "lane": "ci",
          "next": [
            {
              "to": "r5",
              "label": "Yes"
            },
            {
              "to": "r3a",
              "label": "No"
            }
          ],
          "uses": [],
          "label": "All green?"
        },
        {
          "id": "r3a",
          "kind": "task",
          "lane": "dev",
          "next": [
            {
              "to": "r3"
            }
          ],
          "pain": {
            "note": "Flaky tests: failures that are not real bugs",
            "level": 2
          },
          "uses": [],
          "label": "Fix the failing tests"
        },
        {
          "id": "r5",
          "kind": "task",
          "lane": "qa",
          "next": [
            {
              "to": "r6"
            }
          ],
          "uses": [],
          "label": "Manual test pass and exploratory testing"
        },
        {
          "id": "r6",
          "kind": "decision",
          "lane": "qa",
          "next": [
            {
              "to": "r7",
              "label": "Clear to ship"
            },
            {
              "to": "r6a",
              "label": "Blockers found"
            }
          ],
          "uses": [],
          "label": "Any blocking bugs?"
        },
        {
          "id": "r6a",
          "kind": "task",
          "lane": "dev",
          "next": [
            {
              "to": "r3"
            }
          ],
          "uses": [],
          "label": "Fix and re-verify"
        },
        {
          "id": "r7",
          "kind": "task",
          "lane": "pm",
          "next": [
            {
              "to": "r8"
            }
          ],
          "uses": [],
          "label": "Go / no-go meeting"
        },
        {
          "id": "r8",
          "kind": "decision",
          "lane": "pm",
          "next": [
            {
              "to": "r9",
              "label": "Go"
            },
            {
              "to": "r8a",
              "label": "No go"
            }
          ],
          "uses": [],
          "label": "Go?"
        },
        {
          "id": "r8a",
          "kind": "task",
          "lane": "pm",
          "next": [
            {
              "to": "r2"
            }
          ],
          "uses": [],
          "label": "Choose a new date and tell everyone"
        },
        {
          "id": "r9",
          "kind": "subprocess",
          "lane": "ci",
          "next": [
            {
              "to": "r10"
            }
          ],
          "uses": [],
          "child": "m_canary",
          "label": "Deploy to 5% of users first"
        },
        {
          "id": "r10",
          "kind": "decision",
          "lane": "ops",
          "next": [
            {
              "to": "r11",
              "label": "Yes"
            },
            {
              "to": "r10a",
              "label": "No"
            }
          ],
          "uses": [],
          "label": "Metrics healthy?"
        },
        {
          "id": "r10a",
          "kind": "task",
          "lane": "ops",
          "next": [
            {
              "to": "r2"
            }
          ],
          "pain": {
            "note": "A rollback at midnight, after a launch nobody could stop",
            "level": 3
          },
          "uses": [],
          "label": "Roll back and alert the team"
        },
        {
          "id": "r11",
          "kind": "task",
          "lane": "ci",
          "next": [
            {
              "to": "r12"
            }
          ],
          "uses": [],
          "label": "Roll out to everyone"
        },
        {
          "id": "r12",
          "kind": "task",
          "lane": "pm",
          "next": [
            {
              "to": "r13"
            }
          ],
          "uses": [],
          "label": "Announce: release notes and email"
        },
        {
          "id": "r13",
          "kind": "end",
          "lane": "cust",
          "next": [],
          "uses": [],
          "label": "Customers see the new feature"
        }
      ],
      "title": "Ship a product release without a 2 a.m. page",
      "parent": null
    },
    "m_canary": {
      "id": "m_canary",
      "lanes": [
        {
          "id": "ci",
          "name": "CI/CD pipeline",
          "type": "system"
        },
        {
          "id": "ops",
          "name": "Operations",
          "type": "person"
        }
      ],
      "steps": [
        {
          "id": "c1",
          "kind": "start",
          "lane": "ci",
          "next": [
            {
              "to": "c2"
            }
          ],
          "uses": [],
          "label": "Route 5% of traffic to the new version"
        },
        {
          "id": "c2",
          "kind": "task",
          "lane": "ops",
          "next": [
            {
              "to": "c3"
            }
          ],
          "uses": [],
          "label": "Watch errors, speed and sign-ups for an hour"
        },
        {
          "id": "c3",
          "kind": "decision",
          "lane": "ops",
          "next": [
            {
              "to": "c4",
              "label": "Yes"
            },
            {
              "to": "c5",
              "label": "No"
            }
          ],
          "uses": [],
          "label": "Worse than the old version?"
        },
        {
          "id": "c4",
          "kind": "task",
          "lane": "ci",
          "next": [
            {
              "to": "c6"
            }
          ],
          "uses": [],
          "label": "Automatic rollback"
        },
        {
          "id": "c5",
          "kind": "task",
          "lane": "ci",
          "next": [
            {
              "to": "c6"
            }
          ],
          "uses": [],
          "label": "Increase to 25%, then 50%"
        },
        {
          "id": "c6",
          "kind": "end",
          "lane": "ops",
          "next": [],
          "uses": [],
          "label": "Decision logged"
        }
      ],
      "title": "Canary deploy",
      "parent": {
        "map": "m_root",
        "step": "r9"
      }
    }
  }
}
```