Did Your Prompt Change Help? Prove It with an Eval

TL;DR Change a prompt or upgrade a model without LLM evals and your users find the regression for you. An eval is 20 to 50 real inputs with expected results, kept in git next to the prompt and run on every change. Use the cheapest grader that works, keep a slice you never tune against, and ship nothing that cannot beat the score.

Do this: Put 20 real inputs and their expected results in git next to your prompt, and score the current prompt before you change it.

Part 2 ended with the test-case generator running the orders suite and reading its own failures, and one question: when you edit its system prompt, is the next version better, or only different?

Here is a change that throws no error. You move the generator from claude-opus-5 to claude-opus-5-5, a newer model at a lower price. Its default effort is medium, where the old one defaulted to high. Every call now reasons less than it did yesterday, and nothing in your logs says so.

This series, in six parts:

1. Your Prompt Worked in Chat. Why Is the LLM API Different?: the call, and what comes back.
2. Build the Agent Loop Yourself, Once: tools, and the loop every agent runs.
3. Did Your Prompt Change Help? Prove It with an Eval (you are here): a number for every change.
4. RAG That Works: Why Retrieval Matters More Than the Prompt: answers from your own docs.
5. Your LLM Demo Works. What Every Call Needs in Production: cost, failures, logs, injection.
6. Your LLM Proxy Retries the Wrong Errors: rate limits, retries and streams.

What an eval is

A set of inputs, the result each should produce (or how to score it), and a script that runs them all and prints a number. Without one, "the prompt got better" is a feeling.

• Start with 20 to 50 real inputs, not 500 invented ones. For the generator: real requirements from your backlog.
• Keep them in git next to the prompt, so a prompt change and its score land in the same review.
• Run them on every change: a prompt edit, a model upgrade, a new effort level.

Eval products exist, but a folder of YAML cases and a pytest runner is a legitimate first version, and it is where I'd start.

Pick the cheapest grader that works

The middle row is the generator's main grader, and it costs nothing but CPU. Reach for a judge only for what code cannot check.

Keep a slice you never tune against

Tune a prompt against the same cases long enough and it learns those cases, not the task. So hold some back.

1. Run the current prompt and record the baseline.
2. Change one thing.
3. Run the tuning set.
4. Run the held-out slice you have never tuned against.
5. Ship only if both beat the baseline.

Upgrades become routine

Back to the hook. Models are deprecated, and behaviour shifts between generations: verbosity, how eagerly they call tools, how literally they read, parameters that disappear. With an eval, an upgrade is a checklist:

1. Pin model IDs. No "latest" aliases in production.
2. Read the provider's migration guide.
3. Run your evals on the new model.
4. Delete the prompt workarounds it no longer needs.
5. Re-tune effort, then ship or wait, with data.

Step 3 catches the hook's silent change; step 5 fixes it. Set effort in code, so the value sits in the diff:

response = client.messages.create(
    model="claude-opus-5-5",
    max_tokens=16000,
    thinking={"type": "adaptive"},              # always on for Opus 5.5; explicit is fine
    output_config={"effort": "high"},           # low | medium | high | xhigh | max
    messages=[...],
)

Try it

Here is my prompt and five real inputs it handles: <paste>.
Draft 20 eval cases in YAML: the five real ones, plus edge cases,
missing-information cases and one adversarial input. For each, say
which grader fits (exact, code assertion or LLM judge) and why.

The generator's cases now compile and pass, and you can prove whether a change helped. What the eval cannot tell you is whether a case tests the rule your product really has. The retry policy for webhook deliveries lives in your docs, not in the model. Part 4, RAG That Works, gives the generator those docs.

Read this with your AI

Read the post "Did Your Prompt Change Help? Prove It with an Eval".
1. Summarise it in 5 bullets for a <your role>.
2. Ask me 3 questions to check I understood; wait, then correct me.
3. Write a 30-line Python runner for YAML eval cases that prints a
   score and lists failing case ids.