Incident response runbook
Detect, command, mitigate, verify
Signals
Incident command
Service mitigation
Recovery evidence
Stakeholder communication
Escalation and rollback
Detect
Triage and mitigate
Verify and close
Command owns the incident
SLO alert
Burn rate
Page on-call
Acknowledge
Triage
Scope impact
Declare
Assign commander
SEV-1/2
Contain
Stop growth
Recover
Restore service
Verify
SLO and traces
15 min stable
Status update
Impact and ETA
Resolve
Final update
Escalate
Specialist
Rollback
Last good
page
still impacted
regressed
diagram.html#The link keeps the view, selection, route and playback.
About this diagram
Steps done by different owners, in order: a release process, an incident runbook, an agent’s tool call.
Ask for one like it
Draw our release process as a workflow, with a lane for rollbacks.
The JSON
{ "kind": "workflow", "title": "Incident response runbook", "subtitle": "Detect, command, mitigate, verify", "lanes": [ { "id": "signals", "label": "Signals" }, { "id": "command", "label": "Incident command" }, { "id": "mitigation", "label": "Service mitigation" }, { "id": "recovery", "label": "Recovery evidence" }, { "id": "communication", "label": "Stakeholder communication" }, { "id": "exceptions", "label": "Escalation and rollback", "tone": "exception" } ], "phases": [ { "id": "detect", "label": "Detect", "nodes": ["alert", "page"] }, { "id": "triage", "label": "Triage and mitigate", "nodes": ["triage", "declare", "contain", "update", "escalate"] }, { "id": "close", "label": "Verify and close", "nodes": ["recover", "verify", "resolve", "rollback"] } ], "groups": [{ "id": "owners", "label": "Command owns the incident" }], "nodes": [ { "id": "alert", "type": "queue", "lane": "signals", "card": { "title": "SLO alert", "subtitle": "Burn rate", "brand": "pagerduty" } }, { "id": "page", "type": "external", "lane": "command", "group": "owners", "card": { "title": "Page on-call", "subtitle": "Acknowledge" } }, { "id": "triage", "type": "service", "lane": "command", "group": "owners", "card": { "title": "Triage", "subtitle": "Scope impact" } }, { "id": "declare", "type": "security", "lane": "command", "group": "owners", "card": { "title": "Declare", "subtitle": "Assign commander", "tag": "SEV-1/2" } }, { "id": "contain", "type": "service", "lane": "mitigation", "card": { "title": "Contain", "subtitle": "Stop growth" } }, { "id": "recover", "type": "gateway", "lane": "mitigation", "card": { "title": "Recover", "subtitle": "Restore service" } }, { "id": "verify", "type": "database", "lane": "recovery", "card": { "title": "Verify", "subtitle": "SLO and traces", "tag": "15 min stable" } }, { "id": "update", "type": "client", "lane": "communication", "card": { "title": "Status update", "subtitle": "Impact and ETA" } }, { "id": "resolve", "type": "external", "lane": "communication", "card": { "title": "Resolve", "subtitle": "Final update" } }, { "id": "escalate", "type": "security", "lane": "exceptions", "card": { "title": "Escalate", "subtitle": "Specialist" } }, { "id": "rollback", "type": "queue", "lane": "exceptions", "card": { "title": "Rollback", "subtitle": "Last good" } } ], "edges": [ { "id": "e1", "from": "alert", "to": "page", "label": "page", "tone": "main" }, { "id": "e2", "from": "page", "to": "triage", "tone": "main" }, { "id": "e3", "from": "triage", "to": "contain", "tone": "main" }, { "id": "e4", "from": "contain", "to": "recover", "tone": "main" }, { "id": "e5", "from": "recover", "to": "verify", "tone": "main" }, { "id": "e6", "from": "verify", "to": "resolve", "tone": "main" }, { "id": "e7", "from": "triage", "to": "declare", "tone": "security" }, { "id": "e8", "from": "declare", "to": "update", "kind": "async" }, { "id": "e9", "from": "update", "to": "escalate", "label": "still impacted", "tone": "security" }, { "id": "e10", "from": "verify", "to": "rollback", "label": "regressed", "tone": "error" } ], "views": [ { "id": "command", "label": "Command", "caption": "A page is not an incident until someone owns command.", "nodes": ["alert", "page", "triage", "declare", "update"] }, { "id": "recovery", "label": "Recovery", "caption": "Mitigation reduces impact; only a stable SLO window proves recovery.", "nodes": ["contain", "recover", "verify", "resolve", "rollback"] } ], "notes": [ { "title": "Ownership first", "items": [ "A page is not an incident until someone owns command", "Severity and scope are explicit before mitigation spreads", "Escalation names the missing expertise" ] }, { "title": "Recovery is evidence", "items": [ "Mitigation can reduce impact without proving recovery", "SLOs and traces must stay healthy for a fixed window", "The final update follows verification, not optimism" ] } ]}