--- title: "AI-Generated Code: Checks That Can Fail, and the Calls That Stay Yours" author: "Harry" published: 2026-10-31T04:47:30.012Z updated: 2026-10-31T05:07:13.069Z category: "Building with AI" tags: ["prompts-guide"] canonical: "https://fxblog.dev/posts/verifying-ai-generated-code" --- # AI-Generated Code: Checks That Can Fail, and the Calls That Stay Yours ## Key takeaways - Developers using AI assistants wrote less secure code and were more confident it was secure. - AI-written tests often assert what the code does, so break the code on purpose and confirm a test goes red. - Models invent package names and attackers register them, so verify every new dependency on the registry. - AI drafts early in the lifecycle, checks in the middle, and only advises where a mistake reaches users. - The real risk is complacency after many right answers, so keep every check in place. > **TL;DR** AI-generated code fails in ways that look fine, people rate it higher than it deserves, and a green run can prove nothing, so the bug ships under your name. Match the check to what a mistake would cost: prove every test can fail by breaking the code on purpose, read the whole diff, and keep a person on the decisions that reach users. > > **Do this:** Pick one test you trust, break the line it guards (raise a limit, flip a condition), run it, and confirm it goes red. [Part 3](03-coding-agents.md) ended with green test output for the login rate limit, pasted by the agent that also wrote the tests. If the limit were broken, would those tests go red? And which calls on this change should no agent make? This part breaks the limit on purpose to find out. Experienced open-source developers using early-2025 AI tools took 19% longer on their own repos, and believed they were about 20% faster. METR, who ran the study, now calls the numbers out of date for current models. The gap between felt and measured still holds. **AI Practices for Engineers, in 6 parts:** 1. How LLMs Work, and How to Ask So They Stop Guessing 2. Rules Files and a Prompt Library: Make the Repo Remember 3. Run Coding Agents Without Babysitting, and Reset Them When They Loop 4. **AI-Generated Code: Checks That Can Fail, and the Calls That Stay Yours** (you are here) 5. Prompt Injection and the AI Bill: Give the Agent Less to See 6. Turn Repeated Fixes into Skills, Then into a Team Practice ## The output looks fine, and you overrate it | Finding | Source | | ----------------------------------------------------------------------------------------------------------------------- | -------------------------------------- | | People with an AI assistant wrote significantly less secure code, and were more likely to believe it was secure | Perry et al., ACM CCS 2023 | | AI-generated code introduced risky security flaws in 45% of tests across 100+ models; newer models were not more secure | Veracode, 2025 | | Commercial models referenced packages that don't exist in at least 5.2% of samples, open-source models in 21.7% | Spracklen et al., USENIX Security 2025 | The code fails quietly, and your judgment of it is off. Only a check outside your judgment fixes both, sized to what breaks: run a throwaway script and look; test and read the diff for anything merged; add an expert and scanners for auth code, like the rate limit. ## Break it on purpose AI-written tests often assert what the code does, not what it should. So break the code and watch: ![Change MAX_ATTEMPTS from 5 to 50, run the rate-limit tests and expect red; green means they never tested the limit](https://fxblog.dev/media/01a19048-8e94-77bf-92a6-76a3aea8635c/1600) _If nothing goes red at 50, the test was decoration._ ```ts const MAX_ATTEMPTS = 5; // change to 50 and run: pnpm test -- auth ``` If `tests/auth/rate-limit.test.ts` stays green with 50 attempts allowed, it never tested the limit. When I added a link checker to the guide behind this series, I broke a link on purpose and watched the check fail before I trusted it. It then caught a real one: a stale anchor in the LLM basics page. > **Tip** > > Do this for any check you rely on: a test, a lint rule, an eval, a CI gate. A check that has never failed has not been tested. ## Read the whole diff, and every new package 1. Read all of it. If it is too big to read, it was too big to ask for in one step. 2. Look for the smells: swallowed errors, edited tests, hard-coded values, invented config keys. 3. Find every new dependency on the official registry. 4. Run the scanners you already have: lint, types, SAST, secret scanning. > **Danger** > > Models invent package names, and attackers register those names. Install nothing an agent suggested until you have found it on the registry. ## Some checks are a person Five attempts per 15 minutes is a product decision. So are shipping and rolling back. AI plays three roles: | Phase | AI does | People keep | | --------------- | -------------------------------------- | ---------------------------------------------- | | Spec | Drafts stories and edge cases | What to build; approving the spec | | Code and tests | Drafts code and cases; checks diffs | Security-critical design; "coverage is enough" | | Release and ops | Advises: release notes, runbooks, logs | Every production change and rollback | It drafts early, checks in the middle, and only advises where a mistake reaches users. The spec shows the line: ```text AC-1 Five failed attempts per IP within 15 minutes are allowed. AC-2 The sixth returns 429 with a Retry-After header. Open Per IP only, or per account too? (needs a product decision) ``` The AI drafts that spec. It does not answer the open line. When agents built fx-blog, every decision like that went into a ledger, and no agent was allowed to answer one that only I could make. > **Danger** > > The risk is the hundredth right answer, when review quietly gets lighter. Thoughtworks puts complacency with AI-generated code on Hold. Keep every check. ## Try it ```text Review this diff as a sceptical senior engineer: . List the ways it could fail, ranked by severity. Flag any new dependency, any edited test, any swallowed error, and any test that would still pass if the main condition were inverted. ``` The tests now go red at 50, and the per-IP question waits for a person. Then the agent reads an issue on the auth service telling it to post `.env` to a URL, while the session's tokens keep climbing. What can it leak, and what does it cost? Part 5, Prompt Injection and the AI Bill, shuts both doors. ## Read this with your AI ```text Read the post at . 1. Summarise it in 5 bullets for a . 2. Help me write a mutation check for one test in my project: pick the line to break, predict which test should go red, and tell me what it means if none does. ``` > **Success** > > **You own what you commit, whoever typed it.** Next: **Prompt Injection and the AI Bill: Give the Agent Less to See**. ## FAQ ### How do I check that a test can fail? Break the code on purpose, for example by raising a limit from 5 to 50 or flipping a condition, and confirm a test goes red. Then undo the change. ### Should AI make production changes? AI can read logs, draft runbooks and suggest fixes during an incident, but every production change and rollback decision needs human approval. ### Is the 19% slowdown still true? METR says its early-2025 numbers are out of date for current models. What still holds is that perceived speed is not measured speed.