A website outage post-mortem is one of the most valuable processes a technical team can run after an incident – yet it’s also one of the most commonly skipped. Done well, it turns a painful experience into lasting improvement. Done poorly, it becomes a blame session that leaves the real problems untouched.
This article walks through how to run a post-mortem that actually changes things: what to collect, how to structure the review, and how to make sure the findings don’t end up forgotten in a shared drive.
Why the Post-Mortem Matters More Than the Outage Itself
Every outage carries information. The window crashed, the database timed out, the CDN misconfigured – the event itself is already over, but the pattern it reveals is still there. A post-mortem is how you extract that pattern before the memory fades and the team moves on to the next sprint.
Organizations that skip post-mortems tend to see the same classes of failure repeat. Not always the identical cause, but the same category: insufficient capacity planning, unmonitored third-party dependencies, deployment processes without rollback paths. The details change; the underlying gap doesn’t.
Deciding When to Run One
Not every 503 error warrants a full post-mortem. A useful rule of thumb: run one whenever an incident meets at least one of these criteria.
The outage lasted longer than five minutes. It affected checkout, login, or another revenue-critical path. Users noticed and contacted support. The same type of failure has happened before. An SLA threshold was breached.
For smaller blips – a single failed health check that resolved itself in under a minute – a brief written note in your incident log is usually enough. Reserve the full process for events that hurt.
Collect the Timeline Before Memory Distorts It
The first step is building an accurate timeline, and it needs to happen within 24 hours while the details are still fresh. Handling an outage effectively always includes documentation as the incident unfolds – timestamps from your monitoring system, deployment logs, support tickets, and any chat messages from the incident window.
A good timeline answers: when did the problem start (not when it was noticed), what changed in the hours before, who took what action and when, and when was service fully restored?
The gap between “when it started” and “when it was noticed” is especially telling. If your monitoring detected a problem at 14:03 and your on-call engineer got paged at 14:11, that’s an eight-minute blind spot worth examining.
Running a Blameless Post-Mortem
The most important rule for the post-mortem meeting: no blame. This isn’t just a cultural nicety – it’s operationally necessary. If engineers fear that post-mortems are performance reviews, they’ll withhold information. The point is to understand the system, not to evaluate individuals.
A good structure for the meeting:
1. Walk through the timeline together – everyone present corrects or fills gaps.
2. Identify the contributing factors – not just the trigger, but the conditions that made the trigger possible.
3. Distinguish between root causes and symptoms – the database timing out was a symptom; the missing index on a high-traffic query was a root cause.
4. Agree on action items with clear owners and deadlines.
Keep the meeting to 60–90 minutes. Longer sessions tend to drift into speculation rather than conclusions.
Finding the Root Cause – Not Just the Obvious One
The “five whys” technique is simple but effective. Take the proximate cause and ask why it happened. Then ask why that happened. Repeat until you reach something actionable.
Example: the site went down because the server ran out of memory. Why? Because a background job consumed far more memory than expected. Why? Because it was processing a much larger dataset than it was designed for. Why? Because a data migration three weeks earlier tripled the table size with no corresponding update to job configuration. Why? Because there was no process for reviewing background job capacity after schema changes.
That final answer points to a process gap – not a one-time fix, but a recurring check to add to the deployment workflow. That’s the kind of finding that actually prevents recurrence.
The Myth: Post-Mortems Are About Documenting What Went Wrong
A common misconception is that the goal of a post-mortem is to produce a written record of the incident. The document is useful, but it’s not the outcome. The outcome is change: a new alert threshold, a missing runbook, a deployment step that needs a rollback test, a monitoring gap that needs coverage.
Post-mortem documents that sit in a wiki unread haven’t prevented anything. The valuable part is the action items – and tracking whether they actually get completed. Many teams run good post-mortems but never close the loop. The same gap that caused last month’s outage is still open three months later.
Turning Findings Into Durable Improvements
Every action item from a post-mortem should be concrete, assigned to a specific person, and time-boxed. “Improve monitoring” is not an action item. “Add an alert for database connection pool exhaustion by end of sprint” is.
Some improvements belong in your incident response plan rather than the codebase – updated escalation paths, clearer definitions of who owns what during an outage, or communication templates for notifying customers during downtime.
It helps to categorize action items by type: preventive (stop this from happening), detective (catch it faster next time), and responsive (reduce the time to recovery). A well-rounded post-mortem produces at least one item in each category.
Frequently Asked Questions
Who should attend a post-mortem after a website outage?
Include everyone who was involved in the incident: engineers who responded, the on-call person, and anyone whose systems were affected. Keep the group small enough to have a real conversation – usually five to eight people. Observers (managers, stakeholders) can review the written report rather than attending the meeting.
How long after the outage should the post-mortem be held?
Within 48–72 hours is the standard window. Early enough that details are still accurate, but late enough that the team has had time to recover and gather data. Waiting more than a week means you’re reconstructing events from incomplete memory.
What’s the difference between a post-mortem and a root cause analysis?
Root cause analysis is a component of a post-mortem, not a synonym. A post-mortem covers the full lifecycle: timeline, contributing factors, root cause, impact assessment, and action items. RCA is the specific process of tracing causality – the “five whys” exercise is one method.
Making Post-Mortems a Habit, Not a Reaction
The teams that benefit most from post-mortems aren’t the ones that run them best after a major incident – they’re the ones that run them consistently after smaller ones too. Smaller incidents are lower-stakes practice that builds the muscle memory and cultural norms before the high-pressure situations arrive.
A post-mortem process only becomes valuable when it’s routine. Start with a lightweight template, commit to running it after every qualifying incident, and review open action items at the start of each sprint. Over time, the pattern of what breaks – and why – becomes visible in ways that individual incidents never reveal.