The Incident Response Framework Your Team Will Actually Use

A practical incident response framework for engineering teams where the "everyone swarms" model has started causing more chaos than it solves.

Every engineering team has an implicit incident response process. Most of them discover it doesn't work during a SEV-1 at 2 AM - three teams involved, customer PII potentially exposed, and nobody can answer a basic question: who is running this?

The problem isn't engineering talent. It's structural. At seed stage, an alert fires, everyone swarms, shared context handles the rest. At Series C+, with multiple teams across time zones and SOC 2 auditors asking for your incident response procedure - the swarm model actively makes things worse.

The cost compounds fast. Ambiguous ownership extends MTTR. Missing audit trails become compliance findings. Undocumented breach timelines become regulatory exposure.

The one principle that changes everything: every decision made during an incident should have been made before the incident started.

The Vocabulary - Get It Right

Severity and Type get confused constantly. Here's the distinction that matters:

Severity dictates response speed, who gets paged, and escalation paths.

Level	Impact	Response	Escalation
SEV-1	Full outage or confirmed data breach. All users affected. Revenue loss active.	Ack in 5 min. All-hands war room.	CTO in 15 min. Status page in 20 min. Legal notified. GDPR 72h / HIPAA 60-day clock starts if personal data involved.
SEV-2	Major degradation. Key feature down. Error budget burning >1%/hour.	Ack in 15 min. Team assembled.	Eng lead notified. Status page in 30 min.
SEV-3	Minor degradation. Workaround exists. <10% users affected.	Respond in 1 hour. On-call owns.	Team lead notified. Internal tracking.
SEV-4	Cosmetic or internal only. No customer impact.	Next business day.	Standard ticket.

Type - Availability, Security, Data Integrity, Third-Party, Change-Induced, or Privacy — determines which playbook runs and whether compliance obligations are triggered. A site outage and a data breach require fundamentally different people, tools, and communication paths.

The cardinal rule: never debate severity during a live incident. Classify fast, adjust later. Customer data exposure → auto SEV-1. Can't classify in 5 minutes → default SEV-2. Error budget >80% consumed → escalate one level.

The War Room: Who Does What

If everyone is debugging, nobody is communicating. If you don't have a defined role, stay out of the war room.

Incident Commander (IC) - Runs the incident end-to-end. Delegates, decides, escalates. Never hands-on-keyboard during SEV-1/2.

Tech Lead - Drives investigation and fix. Their team is the only one touching production. No freelancing.

Comms Lead - Owns status page and internal updates. Primary job: keeping everyone else out of the tech team's way.

Scribe - Real-time timeline of decisions and timestamps. This becomes your audit trail.

The IC doesn't debug. The Tech Lead doesn't communicate externally. The Comms Lead doesn't touch production. That separation is what turns a 90-minute mess into a 30-minute resolution.

Four Principles That Separate Fast Resolution From Extended Chaos

1. Containment Before Root Cause

Your customers don't care why it broke. During a live incident, containment is your only goal. Roll back the deploy, kill the feature flag, reroute the traffic, or fail over to a vendor. Investigate later.

2. Single-Threaded Production Access

Three engineers applying contradictory hotfixes simultaneously is how you turn one incident into three. Everything goes through the IC. If you aren't the designated Tech Lead, you don't touch production.

3. Security Incidents Override Standard Process

When a security incident is declared, normal process gets overridden. Keep it need-to-know until forensics confirm the vector is external. Use a codename for the incident channel. Prefix written comms with "Attorney Work Product" to protect legal privilege. Snapshot logs and system state before anyone starts remediating — evidence destroyed during containment is evidence you can't use during litigation.

4. Blameless Postmortems, Accountable Follow-Through

When the fire is out, the postmortem begins. Assume good intentions and focus on the systems and process gaps that allowed the failure. Use a strict "Because → Why" chain to drill past the immediate technical trigger. Unresolved postmortem items are the #1 predictor of repeat incidents. Target >90% on-time completion. P1 item overdue by a week? It gets escalated to the CTO.

The Template

Incident response is about decision speed. One commander, one fixing team, one communication channel. Every minute spent figuring out who is doing what is a minute your users are waiting.

We open-sourced a complete Incident Response Template — mapped to SOC 2 Type II, ISO 27001, HIPAA, and GDPR. Severity matrices, role definitions, communication templates, breach notification timelines, on-call design, and postmortem structures. Everything an auditor will ask for and everything your team actually needs at 2 AM.

[Get the template →]