When You Break Something: Owning a Mistake Without Losing Trust

TL;DR When you break production, the instinct is to fix it quietly before anyone notices. Customers keep failing while you try, and the hiding is what the team remembers. What to do when you break production: say it in the team channel within minutes, roll back before you debug, then write a short incident note that blames the gap in the system, not you.

Do this: Before you need it, find out how your team rolls back a deploy and where problems get reported.

Part 3 ended at 17:38 on a Friday, with Pip's discount feature six minutes into production and checkout returning errors to real customers. The error-rate graph on her screen is a cliff. For about ten seconds she considers fixing it before anyone notices.

Then she stands up. "Checkout is down! I think it's mine!"

Dev Voyage: Calm Sea, in five parts:

1. Your First 90 Days: How to Join a Crew Without Sinking
2. Ask Better Questions: The Skill That Saves Your First Month
3. Make Friends on the Ship: Working Happily With Colleagues
4. When You Break Something: Owning a Mistake Without Losing Trust (you are here)
5. Pass Probation on Purpose

Say it before you fix it

Shouting was the right call, and the written version matters just as much, because it's what people read when they open the channel.

Heads up: checkout is returning 500s since about 17:38.
It started right after my deploy of the discount change (17:32).
Rolling it back now. Will update here in 10 minutes.

What broke, since when, the likely cause, what she's doing, when she'll speak next. No three-paragraph apology, no guessing about blame.

Stop the damage, then look for the cause

Kit didn't ask why it broke. Not yet. With customers failing, there's an order:

1. Stop it: roll back, turn off the feature flag, or move traffic away. The fastest safe option wins, elegant or not.
2. Watch the error rate until you're sure it stopped.
3. Only then, with users safe, start finding out why.

Debugging a live fire while checkout keeps failing is how a short incident becomes a long one. Pip's rollback took eleven minutes.

A note that blames the gap, not the person

On Monday Pip wrote her first incident note. Short and true beats long and polished.

# Checkout returned 500s for 13 minutes

What happened: the discount change read a column the production
database did not have yet. Checkout failed on every order.
Impact: checkout errors from 17:38 to 17:51, Friday 6 Nov.
Timeline:
- 17:32 discount change deployed
- 17:38 first 500 errors on checkout
- 17:40 reported in #red-fox-team, rollback started
- 17:51 rollback done, errors stopped
Root cause: the migration ran after the code that needs it.
Nothing in CI checks the order.
What we'll change:
- Dry-run migrations in CI before deploy (owner: Pip)
- Add "migration first" to the deploy checklist (owner: Brock)

Read it again and look for "Pip forgot". It isn't there. The cause is written as a fact about the pipeline, because that's the part anyone can change.

This is what Google's SRE book calls a blameless postmortem: the review looks for the conditions that let a mistake through, not for someone to punish. The reason is practical. People who get blamed stop reporting early, and the next incident stays hidden longer.

Pip owning the CI fix isn't a punishment. She knows the failure best, and it turned her worst Friday into her first change to how the whole crew ships.

One apology, then keep sailing

Say sorry once, short and pointing forward: "Sorry, my deploy broke checkout for 13 minutes. I've added a CI check so a missing migration can't ship again."

If you lead a team, your first reply sets the tone for every incident after it. Kit's "thanks for shouting fast" is worth copying word for word.

Try it

Before your first incident, not during it:

1. Ask how to roll back a deploy on your team.
2. Find the channel where problems get reported.
3. Read one of your team's past incident notes, and save its format.

Pip's note was read by the whole crew, and Kit thanked her for it in the channel. Then, on Monday afternoon, she noticed a calendar invite: her probation review, in three weeks. Friday is the freshest thing in everyone's memory, and she has no idea what Kit will write. Part 5, Pass Probation on Purpose, is how she stops waiting for a verdict.

Read this with your AI

Read the post "When You Break Something: Owning a Mistake Without Losing Trust".
1. Write the message I would post in my team channel in the first two minutes of an incident.
2. I'll describe a past mistake. Rewrite its root cause as a fact about the system, not a person.
3. Suggest one check my team could add so it cannot happen the same way again.