The First Thirty Minutes of a Production Incident

The first thirty minutes of a production incident determine whether the postmortem three days later gets written from evidence or from memory. Most teams burn that window doing the wrong things in the wrong order. They argue severity in a Slack thread, wait for the one engineer who actually understands the billing service to wake up and check their phone, or jump straight into remediation before anyone thinks to capture what the system looked like at the moment it broke. By the time someone remembers to pull logs, the retention window has already rolled them off a cliff, and the incident review turns into a room full of people trying to reconstruct a crime scene from what they half-remember instead of what actually happened.

I've sat through both kinds of postmortems, the ones with a timeline built from screenshots and the ones built from someone saying "I think it was around 2:15, maybe 2:20." The difference in how those meetings go is not subtle.

This isn't a documentation problem. It's a sequencing problem, and on-call engineers get drilled on almost everything except the order these three things need to happen in.

Declare severity first, and don't relitigate it

The biggest time sink in early incident response, hands down, is arguing about how bad something is while it keeps getting worse. Google's SRE book gets at this, and PagerDuty and Atlassian have both built incident command frameworks around a version of the same rule: somebody declares severity in the first few minutes, using criteria that were written down before the incident, and nobody argues about it in the moment. You fix the call later if it was wrong, but you don't spend ten minutes getting everyone to agree.

Four tiers work better than seven, mostly because a scale needs to be usable by someone half-asleep at 3 a.m. SEV1 is customer-facing at scale, data loss, or a security hole; the org needs to know now. SEV2 is real degradation but contained, one region, one feature, a slice of customers. SEV3 is internal breakage nobody outside the building will notice, and SEV4 is a heads-up, not yet an incident. Whoever spots the problem first says the severity out loud in the channel, within two or three minutes, no committee required. If they're wrong, someone with more context corrects it later, and that correction takes ten seconds. Compare that to the twenty minutes teams routinely lose debating whether something counts as a "real" SEV1. Twenty minutes doesn't sound like much until it's the twenty minutes where the database connection pool finishes exhausting itself.

There's a second reason the declaration matters: it's supposed to trigger something, whether that's paging a second responder, updating a status page, or notifying whoever needs notifying. If typing "SEV1" into a channel doesn't actually do anything, the severity scale is decoration, not infrastructure.

The person running the incident shouldn't be your best engineer

Incident command as a discipline came out of wildfire response in the 1970s. Multiple U.S. agencies kept discovering that uncoordinated firefighting across jurisdictions got people killed, which is how the Incident Command System got formalized. The insight carries over to software almost without translation: the person coordinating response and the person doing the deepest technical work need to be different people, because those two jobs fight over the same attention.

An incident commander's job is deliberately unglamorous: track who's looking at what, decide when to pull in another team, own the updates going out to stakeholders, and call it resolved when it's resolved. They are not supposed to be three levels deep in a stack trace. The moment your most senior database engineer is running the fix and fielding four Slack DMs asking for status at the same time, you've lost your sharpest diagnostic resource to context-switching, and you lost it for free.

On small teams, one person often plays both roles for the first five or ten minutes before backup shows up, which is fine as a stopgap, but it stops being fine at minute fifteen. Nobody said "I'm commander, you're investigating," so decisions get made twice, or not at all, because everybody assumed somebody else was keeping track.

Most of your evidence has a shorter half-life than your retention policy

This is the step that gets skipped most, and it's the one that costs the most three days later. Runtime state decays fast, faster than most people expect. Container logs on ephemeral compute might make it through a restart; they might not. In-memory state disappears the instant a process crashes or a pod gets rescheduled. Metrics get downsampled after a window that's usually hours, sometimes a couple of days, rarely weeks, depending on what tier of your observability vendor you're paying for. Distributed tracing tools like Datadog or Honeycomb typically sample a fraction of requests under normal load, which means the one request that kicked off your cascading failure may never have been captured to begin with.

So someone on the response needs a job that is, explicitly, "capture evidence," and it has to be a different job from "fix the problem." It starts in minute five, not after the fire's out. Screenshot the dashboard before the graph ages off retention, and pull raw logs somewhere durable, a ticket, a shared doc, an S3 bucket, anywhere that outlives the default retention window, instead of assuming you can just query for them again next Tuesday. Write down the exact error strings, the exact timestamps, the exact deploy or config state at the moment things broke. "We rolled back and it got better" is not evidence; it's a correlation you'll be arguing about later with nothing to back either side.

Etsy wrote about this in their postmortem culture work years back, and it's still true: a blameless postmortem is only as good as the timeline it's built on, and a timeline reconstructed five days later from memory is fiction wearing analysis as a costume. The teams running good retrospectives aren't the ones with sharp memories. They're the ones who treated evidence capture as real work during the incident, not an afterthought once the pager stopped going off.

None of this is hard, and any engineer can do each of these on their own: declaring a severity, naming a commander, grabbing a screenshot before it's gone. The failure isn't ignorance; it's timing. Teams that skip straight to fixing things end up either escalating too slowly, because nobody flagged how bad it actually was, or explaining to leadership why they heard about a SEV1 from an angry customer instead of from engineering. Teams that name a commander late end up with three people quietly trying the same fix, unaware of each other. Teams that skip evidence capture end up with "root cause: unclear" in the doc and the same failure again six weeks later, because nobody pinned down the mechanism while the data still existed to pin it down with.

A checklist, a bot that asks for severity the second an incident channel spins up, a weekly rotation for who's on commander duty, the habit of screenshotting before you touch anything: none of it costs much, and most teams still won't bother until the incident that makes them wish they had.