How I Built a One-Day Incident Review Process That Drives Real Change

Introduction

Outages happen, and the value they provide depends on how we capture the lessons they teach. In a small to mid-size engineering group the temptation is either to ignore the event or to launch a heavyweight post-mortem that consumes days of meetings and endless documentation. Both approaches waste time and dilute learning.

I have settled on a process that fits into a single workday, produces a concise, actionable document, and forces the team to close the loop on every action item. The process is deliberately lightweight, but it still respects the three pillars of a good review: factual timeline, clear cause analysis, and accountable follow-up. Below I walk through each step, the tools I rely on, and the habits that keep the system from slipping back into blame-oriented or ignored post-mortems.

Why a lightweight approach works

• Speed preserves context. When an incident ends, the details are fresh. A fast turnaround prevents the loss of mental models that would otherwise fade after a weekend.
• Low overhead encourages repetition. If the process feels like a burden, teams will skip it after the first few incidents. A ten-minute preparation and a forty-minute meeting fit comfortably into sprint cycles.
• Focused output drives action. A short document that lists concrete owners and deadlines is easier to scan and act upon than a ten-page narrative.

Research on small technical teams emphasizes that “clear timelines, contributing factors, and action owners matter more than perfect documentation” [1]. The same article notes that value is realized only when follow-up actions are tracked to completion. The lightweight method I describe satisfies both observations.

Core principles

1. Blamelessness is a mindset, not a checklist. The goal is to understand systemic gaps, not to point fingers.
2. Learning over reporting. The output is a set of insights the team can apply tomorrow, not a historical archive.
3. Ownership with deadlines. Every action item has a single owner and a concrete target date.
4. Repeatability. The same template is used for every incident, regardless of size, to build muscle memory.

These principles guide the entire workflow, from data collection to the final review meeting.

The review workflow

1. Immediate capture (0-15 minutes)

As soon as the fire is out, I create a new incident markdown file in our shared repository. The name follows the pattern incident-YYYYMMDD-brief-description.md. The file starts with a pre-filled header:

**Date:** YYYY-MM-DD 
**Duration:** 0h 45m 
**Service(s) impacted:** payments-api, checkout-frontend 
**Severity:** S2 (customer-visible) 

### Timeline (UTC)

- 10:12 - Alert triggered (PagerDuty)
- 10:14 - Primary on-call acknowledges
- …
- 10:55 - Service restored

I populate the timeline in real time from the incident channel; if the channel is still active, I copy the timestamps directly. This eliminates the need for a later reconstruction exercise.

2. Post-incident data pull (15-45 minutes)

Once the incident is stable, I gather the following artifacts:

I paste links to each artifact in a “References” section of the markdown file. This practice, recommended by the Art of CTO guide, keeps the review self-contained and avoids hunting for data later.

3. Root-cause and contributing factors (45-75 minutes)

With the raw data at hand, I draft two short narrative blocks:

• Root cause, a single sentence that pinpoints the immediate technical failure.
• Contributing factors, a bullet list of organizational or process gaps that allowed the root cause to manifest.

Example:

Root cause: A mis-configured feature flag disabled the circuit-breaker for the payment gateway, causing all requests to time out. Contributing factors

• Lack of automated test for feature-flag boundaries.
• No peer-review checklist item for flag roll-outs.
• Inconsistent monitoring of flag state across environments.

The distinction mirrors guidance from several incident-review resources, which stress that focusing on a single root cause often hides deeper systemic issues [4].

4. What went well / what went poorly (75-90 minutes)

I ask the responders to add brief bullet points under two headings. This encourages balanced reflection and prevents the meeting from devolving into a blame session.

### What went well

- The on-call rotation responded within the SLA.
- Alert routing to the correct Slack channel worked flawlessly.

### What went poorly

- The post-mortem template was not pre-filled, causing a delay in timeline entry.
- No test existed for the new feature flag.

5. Action items (90-105 minutes)

Each action item follows a strict format:

- [ ] <Owner>, <Verb> <What> by <Date>, <Metric for verification>

Example:

- [ ] Priya, Add automated integration test for the payment-gateway feature flag by 2026-09-30, Test passes in CI for all environments.

Assigning a metric (for example “test passes in CI”) makes the item verifiable, addressing a common failure mode identified by Codably: actions without a mechanism never get done [3].

6. Review meeting (105-135 minutes)

The meeting is limited to 30 minutes, run by the incident commander (usually the primary on-call). The agenda is fixed:

1. Quick read-through of the timeline (2 minutes).
2. Presentation of root cause and contributing factors (5 minutes).
3. Walkthrough of action items, confirming owners and dates (10 minutes).
4. Open floor for any additional observations (13 minutes).

I enforce a strict timebox on each segment using a shared timer. The meeting is recorded for reference, but the written document remains the single source of truth.

7. Follow-up and tracking (ongoing)

All action items are entered into our project-tracking board (e.g., Jira) as “Incident-<ID>” tickets with the same owner and due date. I set a recurring “Incident Review” reminder in the team calendar to audit open tickets every sprint. When a ticket closes, the reviewer adds a one-sentence note indicating the verification result. This closing loop ensures that the post-mortem does not become a static diary entry, a pitfall highlighted by multiple sources [3][5].

Tools and templates that keep the process lightweight

• Markdown repository, Storing incidents as plain markdown files in a Git repo gives us version control, searchable history, and the ability to review changes over time.
• The Art of CTO post-mortem template, Provides a clean structure that can be copied into a new file with a single CLI command.
• PagerDuty webhook, Auto-populates the “Alert payload” section, reducing manual copy-paste.
• Slack reminder bot, Posts a one-line prompt in the incident channel after resolution: “Please update the incident file with timeline and artifacts.”

These automations keep the human effort to a minimum while preserving the essential information.

Common pitfalls and how to avoid them

When these signals appear, I pause the next review and adjust the template or the meeting cadence accordingly. The process is intentionally iterative; the goal is continuous improvement, not static perfection.

Measuring success

Two simple metrics tell me whether the lightweight review is delivering:

1. Mean time to close action items, Average days from incident to ticket closure. A target of less than ten days indicates that the loop is tight.
2. Recurrence rate of similar incidents, Count of incidents sharing the same primary failure mode over a quarter. A downward trend shows that learning is being applied.

Both metrics are tracked in a quarterly dashboard and discussed during the engineering retrospective. Seeing numbers improve reinforces the habit of performing reviews promptly.

Conclusion

A lightweight incident review does not sacrifice depth for speed. By capturing the timeline in real time, separating root cause from contributing factors, assigning clear owners with verification metrics, and integrating the output into our existing task board, the team gains actionable insight without the drag of a heavyweight process.

The key is discipline: a short, structured markdown file, a 30-minute meeting, and a simple follow-up loop. When the process is repeatable, it becomes a habit, and each outage turns into a catalyst for stronger engineering practices.

// share