---
title: "Did Your Prompt Change Help? Prove It with an Eval"
author: "Harry"
published: 2026-10-31T04:48:09.237Z
updated: 2026-10-31T05:07:29.941Z
category: "Building with AI"
tags: ["prompts-guide"]
canonical: "https://fxblog.dev/posts/llm-evals-prove-prompt-changes"
---

# Did Your Prompt Change Help? Prove It with an Eval

## Key takeaways

- Without an eval, a better prompt is a feeling; with one it is a number.
- Start with 20 to 50 real inputs and expected results kept in git next to the prompt.
- Use the cheapest grader that works: exact match, a code assertion, or a calibrated LLM judge.
- Keep a held-out slice you never tune against, so the prompt does not overfit the eval.
- Moving to claude-opus-5-5 drops the default effort from high to medium; set effort in code and let the eval show the difference.

> **TL;DR** Change a prompt or upgrade a model without LLM evals and your users find the regression for you. An eval is 20 to 50 real inputs with expected results, kept in git next to the prompt and run on every change. Use the cheapest grader that works, keep a slice you never tune against, and ship nothing that cannot beat the score.
>
> **Do this:** Put 20 real inputs and their expected results in git next to your prompt, and score the current prompt before you change it.

[Part 2](02-agent-loop.md) ended with the test-case generator running the orders suite and reading its own failures, and one question: when you edit its system prompt, is the next version better, or only different?

Here is a change that throws no error. You move the generator from `claude-opus-5` to `claude-opus-5-5`, a newer model at a lower price. Its default effort is `medium`, where the old one defaulted to `high`. Every call now reasons less than it did yesterday, and nothing in your logs says so.

**This series, in six parts:**

1. **Your Prompt Worked in Chat. Why Is the LLM API Different?**: the call, and what comes back.
2. **Build the Agent Loop Yourself, Once**: tools, and the loop every agent runs.
3. **Did Your Prompt Change Help? Prove It with an Eval** (you are here): a number for every change.
4. **RAG That Works: Why Retrieval Matters More Than the Prompt**: answers from your own docs.
5. **Your LLM Demo Works. What Every Call Needs in Production**: cost, failures, logs, injection.
6. **Your LLM Proxy Retries the Wrong Errors**: rate limits, retries and streams.

## What an eval is

A set of inputs, the result each should produce (or how to score it), and a script that runs them all and prints a number. Without one, "the prompt got better" is a feeling.

- **Start with 20 to 50 real inputs**, not 500 invented ones. For the generator: real requirements from your backlog.
- **Keep them in git next to the prompt**, so a prompt change and its score land in the same review.
- **Run them on every change**: a prompt edit, a model upgrade, a new effort level.

Eval products exist, but a folder of YAML cases and a pytest runner is a legitimate first version, and it is where I'd start.

> **Tip**
>
> The rule that makes evals pay: a prompt change that can't beat the eval doesn't ship.

## Pick the cheapest grader that works

| Grader                | Use when                              | For the generator                             |
| --------------------- | ------------------------------------- | --------------------------------------------- |
| Exact or schema match | The answer has one right form         | Extraction, classification                    |
| Code assertion        | You can test the output by running it | Does the generated test compile and pass?     |
| LLM as judge          | Quality is fuzzy                      | Does the case test what the requirement says? |

The middle row is the generator's main grader, and it costs nothing but CPU. Reach for a judge only for what code cannot check.

> **Warning**
>
> A judge needs calibration before you trust it. Use a strong model, give it a rubric, and check its scores against a few cases a person scored first.

## Keep a slice you never tune against

Tune a prompt against the same cases long enough and it learns those cases, not the task. So hold some back.

![The eval loop: change one thing, run the eval, compare the score with the baseline on the tuning set, check the held-out set, and ship only if both improve](https://fxblog.dev/media/01a19049-0fec-746b-bce6-06976e461b5f/1600)

_Never tune against the held-out slice._

1. Run the current prompt and record the baseline.
2. Change one thing.
3. Run the tuning set.
4. Run the held-out slice you have never tuned against.
5. Ship only if both beat the baseline.

## Upgrades become routine

Back to the hook. Models are deprecated, and behaviour shifts between generations: verbosity, how eagerly they call tools, how literally they read, parameters that disappear. With an eval, an upgrade is a checklist:

1. Pin model IDs. No "latest" aliases in production.
2. Read the provider's migration guide.
3. Run your evals on the new model.
4. Delete the prompt workarounds it no longer needs.
5. Re-tune effort, then ship or wait, with data.

Step 3 catches the hook's silent change; step 5 fixes it. Set effort in code, so the value sits in the diff:

```python
response = client.messages.create(
    model="claude-opus-5-5",
    max_tokens=16000,
    thinking={"type": "adaptive"},              # always on for Opus 5.5; explicit is fine
    output_config={"effort": "high"},           # low | medium | high | xhigh | max
    messages=[...],
)
```

> **Note**
>
> Prompts are per-model artifacts. Commit `prompt.md` and `model: claude-opus-5-5` in the same commit, so you always know which prompt was tuned for which model.

## Try it

```text
Here is my prompt and five real inputs it handles: <paste>.
Draft 20 eval cases in YAML: the five real ones, plus edge cases,
missing-information cases and one adversarial input. For each, say
which grader fits (exact, code assertion or LLM judge) and why.
```

The generator's cases now compile and pass, and you can prove whether a change helped. What the eval cannot tell you is whether a case tests the rule your product really has. The retry policy for webhook deliveries lives in your docs, not in the model. Part 4, RAG That Works, gives the generator those docs.

## Read this with your AI

```text
Read the post "Did Your Prompt Change Help? Prove It with an Eval".
1. Summarise it in 5 bullets for a <your role>.
2. Ask me 3 questions to check I understood; wait, then correct me.
3. Write a 30-line Python runner for YAML eval cases that prints a
   score and lists failing case ids.
```

> **Success**
>
> You can say with a number whether a change helped. Next: **RAG That Works: Why Retrieval Matters More Than the Prompt**.

## FAQ

### How many eval cases do I need?

Twenty to fifty real inputs are enough to start. Add a case every time a real failure shows up.

### Can an LLM grade another LLM?

Yes for fuzzy quality, if a strong model uses a rubric and you first check its scores against a few cases a person scored.