--- title: "When You Break Something: Owning a Mistake Without Losing Trust" author: "Harry" published: 2026-10-31T04:49:28.394Z updated: 2026-10-31T05:08:08.930Z category: "Career & Teams" tags: [] canonical: "https://fxblog.dev/posts/what-to-do-when-you-break-production" --- # When You Break Something: Owning a Mistake Without Losing Trust ## Key takeaways - Tell the team within minutes, in the channel: what broke, since when, the likely cause, what you are doing and when you will update. - Stop the damage first, usually by rolling back, and look for the cause after users are safe. - An incident note says what happened, the impact, a timeline and what will change, without blame. - A blameless review asks how the system let the mistake through, and adds a check so it cannot happen the same way again. - Apologise once, briefly and pointing forward; then keep asking questions and shipping small things. > **TL;DR** When you break production, the instinct is to fix it quietly before anyone notices. Customers keep failing while you try, and the hiding is what the team remembers. What to do when you break production: say it in the team channel within minutes, roll back before you debug, then write a short incident note that blames the gap in the system, not you. > > **Do this:** Before you need it, find out how your team rolls back a deploy and where problems get reported. Part 3 ended at 17:38 on a Friday, with Pip's discount feature six minutes into production and checkout returning errors to real customers. The error-rate graph on her screen is a cliff. For about ten seconds she considers fixing it before anyone notices. Then she stands up. "Checkout is down! I think it's mine!" **Dev Voyage: Calm Sea, in five parts:** 1. **Your First 90 Days: How to Join a Crew Without Sinking** 2. **Ask Better Questions: The Skill That Saves Your First Month** 3. **Make Friends on the Ship: Working Happily With Colleagues** 4. **When You Break Something: Owning a Mistake Without Losing Trust** (you are here) 5. **Pass Probation on Purpose** ## Say it before you fix it Shouting was the right call, and the written version matters just as much, because it's what people read when they open the channel. ```text Heads up: checkout is returning 500s since about 17:38. It started right after my deploy of the discount change (17:32). Rolling it back now. Will update here in 10 minutes. ``` What broke, since when, the likely cause, what she's doing, when she'll speak next. No three-paragraph apology, no guessing about blame. > **Note** > > **Kit:** Thanks for shouting fast. Roll back. I'm here. > **Pip:** Rolled back. Errors stopped at 17:51. > **Kit:** Go home. Monday: how, not who. ## Stop the damage, then look for the cause Kit didn't ask why it broke. Not yet. With customers failing, there's an order: 1. Stop it: roll back, turn off the feature flag, or move traffic away. The fastest safe option wins, elegant or not. 2. Watch the error rate until you're sure it stopped. 3. Only then, with users safe, start finding out why. Debugging a live fire while checkout keeps failing is how a short incident becomes a long one. Pip's rollback took eleven minutes. ## A note that blames the gap, not the person On Monday Pip wrote her first incident note. Short and true beats long and polished. ```markdown # Checkout returned 500s for 13 minutes What happened: the discount change read a column the production database did not have yet. Checkout failed on every order. Impact: checkout errors from 17:38 to 17:51, Friday 6 Nov. Timeline: - 17:32 discount change deployed - 17:38 first 500 errors on checkout - 17:40 reported in #red-fox-team, rollback started - 17:51 rollback done, errors stopped Root cause: the migration ran after the code that needs it. Nothing in CI checks the order. What we'll change: - Dry-run migrations in CI before deploy (owner: Pip) - Add "migration first" to the deploy checklist (owner: Brock) ``` Read it again and look for "Pip forgot". It isn't there. The cause is written as a fact about the pipeline, because that's the part anyone can change. This is what Google's SRE book calls a blameless postmortem: the review looks for the conditions that let a mistake through, not for someone to punish. The reason is practical. People who get blamed stop reporting early, and the next incident stays hidden longer. ![An incident from Pip's deploy: she rolls back and tells the team, writes an incident note, and the blameless review asks why the system let it ship and adds a CI check](https://fxblog.dev/media/01a1904a-6fb2-7365-b0bc-22ed18e2a261/1600) _From a Friday mistake to a check that stops it next time_ Pip owning the CI fix isn't a punishment. She knows the failure best, and it turned her worst Friday into her first change to how the whole crew ships. ## One apology, then keep sailing Say sorry once, short and pointing forward: "Sorry, my deploy broke checkout for 13 minutes. I've added a CI check so a missing migration can't ship again." > **Warning** > > Don't apologise for paragraphs, write "I was careless" in the note, or go quiet for a week. The first makes the crew comfort you instead of fixing the pipeline. The second is a cause nobody can fix. The third looks like you've given up. If you lead a team, your first reply sets the tone for every incident after it. Kit's "thanks for shouting fast" is worth copying word for word. ## Try it Before your first incident, not during it: 1. Ask how to roll back a deploy on your team. 2. Find the channel where problems get reported. 3. Read one of your team's past incident notes, and save its format. Pip's note was read by the whole crew, and Kit thanked her for it in the channel. Then, on Monday afternoon, she noticed a calendar invite: her probation review, in three weeks. Friday is the freshest thing in everyone's memory, and she has no idea what Kit will write. Part 5, **Pass Probation on Purpose**, is how she stops waiting for a verdict. ## Read this with your AI ```text Read the post "When You Break Something: Owning a Mistake Without Losing Trust". 1. Write the message I would post in my team channel in the first two minutes of an incident. 2. I'll describe a past mistake. Rewrite its root cause as a fact about the system, not a person. 3. Suggest one check my team could add so it cannot happen the same way again. ``` > **Success** > > You know what to do in the first hour after you break something, and how to turn it into a better ship. Next: **Pass Probation on Purpose**. ## FAQ ### Will I get fired for breaking production? In a healthy team, an honest mistake that you report quickly and help fix is not treated as a firing matter; hiding it is what damages trust. Blameless postmortems exist so that people report problems early. If your team punishes honest mistakes, that is worth knowing about the team. ### What goes in an incident report? What happened, who and what was affected, a timeline with times, what stopped it, the root cause, and the changes that will prevent it, each with an owner. Keep people's names out of the cause.