Skip to content
fx-blog

Your Load Test Passed. Did It Measure the Server?

A load test can go green while it measures a saturated generator or a closed loop waiting on a frozen server. ai-perf runs the load in code, judges every run into one of four outcomes, and says NOT EVALUABLE when a run cannot be judged.

Harry4 min readBeginner
  • Judgment
  • Verification
MarkdownSave

Key takeaways

  • A closed loop of users passed a server frozen 3 seconds in every 10 at p95 56 ms, while users arriving at a fixed rate saw p95 2588 ms.
  • In ai-perf a deterministic controller runs the load and the AI is woken by events only, so a long run survives the AI session closing.
  • Every ai-perf run ends PASS (exit 0), WARN (exit 0), FAIL (exit 1) or NOT EVALUABLE (exit 3).
  • A load generator above 80 % CPU in the measured window makes an ai-perf run NOT EVALUABLE instead of a pass.
  • Even a PASS from ai-perf lists what it rests on, such as a missing baseline, an unmonitored target or SLOs nobody agreed.
Contents

TL;DR Load test results can go green while they measure your load generator, or a loop of users politely waiting for a frozen server, instead of the server itself. ai-perf is a performance-testing flow for Claude Code where code runs the load and every verdict says what it rests on. Only here: a load generator above 80 % CPU makes the run NOT EVALUABLE, exit code 3, never a quiet pass.

Do this: Clone ai-perf, run the quickstart, and read the five notes printed under its PASS before you trust any of your own load test results.

A server freezes for 3 seconds in every 10. You point a load test at it, with a fixed pool of users who each wait for an answer before sending the next request. The report says p95 56 ms. PASS.

Sent the same way users really arrive, whatever the server is doing, the p95 is 2588 ms. My own first version of ai-perf gave that first answer. Every engine ran a closed loop, so a stalling server got fewer requests and looked fast.

This series, in three parts:

  1. Your Load Test Passed. Did It Measure the Server? (you are here)
  2. Where Does It Break? Find the Knee, Then the Cause
  3. A Performance Gate in CI That Doesn't Fail on Noise

Why green load test results can lie

Most load test results answer one question: did the numbers stay under a line? Nobody asks what produced the numbers.

What went wrong What the report still shows
The load generator ran out of CPU and under-sent a percentile
A worker died and took its share of the load with it a percentile
The SLO was a default nobody agreed a green tick

In each case the number describes the tool, not your system.

Four questions to ask of any load test

  1. Was the generator healthy? A busy generator timestamps late and sends less.
  2. Did users arrive the way real ones do? A closed loop hides stalls; that is coordinated omission.
  3. Who agreed the SLO? A default threshold is a guess with a decimal point.
  4. Can it say "I can't judge this"? If the only answers are green and red, it is guessing.

How ai-perf judges a run

I kept the AI out of the run on purpose. A soak runs for hours; an AI polling it would cost tokens for every minute, react too late to stop a runaway test, and lose the run when the session closed.

So a deterministic controller starts the workers, merges their latency histograms (never an average of p95s) and stops a runaway run. The AI is woken by events only. Around the run, ten slash commands, from /perf-discover to /perf-report, each write a file, and the next step refuses to start until that file passes its gate.

The ai-perf showcase at step 10 of 10 on the sample shop-api: /perf-report prints VERDICT: WARN, 200 users held with 1.1× headroom over a planned peak of 180

One real run: PASS, with five notes

The repo ships a demo on a built-in fake engine, which needs no target. I ran it as the README says:

textnode bin/perf.mjs report --root examples/quickstart --slug demo --bootstrap
VERDICT: PASS
  note: BOOTSTRAP: no baseline exists; judged on SLOs only. Promote this run to start regression checks.
  note: auto-stop: none was planned, so the default applied (error rate above 50 % for 30 s); plan rules from the SLA in the run spec
  note: target not monitored: a bottleneck can be located only from response times
  note: environment parity: laptop mock; numbers are indicative for production
  note: SLOs not agreed with an owner (source imported/assumed/default): listItems, getItem, createOrder
  summary: At 20 users (100 requests/s), every planned transaction met its SLO.

Five notes under one PASS. A PASS with five notes is weaker than a PASS with none, and the verdict says so instead of hiding it.

Get it, and what it does not do

Try it
git clone https://gitlab.com/agentic-fx/ai-perf.git && cd ai-perf
node bin/perf.mjs doctor

It needs Node 20 or later; Locust needs Python with pip install locust, k6 needs the k6 binary. ai-perf is not in the npx agentic-fx installer yet. It runs Locust, k6, a WebSocket engine and the fake one; Gatling and JMeter suites are imported and audited, not run. Browser and mobile performance are out of scope.

Try it

Try it
node bin/perf.mjs run examples/quickstart/fake.json --root examples/quickstart --wait
node bin/perf.mjs report --root examples/quickstart --slug demo --bootstrap

The shop-api WARN above says it holds 200 users with 1.1× headroom, but not where between 200 and 400 users it broke, or what it was waiting on. Part 2, Where Does It Break? Find the Knee, Then the Cause, reads that staircase step by step and names the full connection pool.

Read this with your AI

textPaste into your AI
Read the post "Your Load Test Passed. Did It Measure the Server?".
1. Summarise it in 5 bullets for an engineer who runs load tests.
2. Ask me 3 questions to check I understood; wait, then correct me.
3. Ask the four questions in the post of my last load test, and tell me which one it fails.

FAQ

Which load tools does ai-perf run?

Locust, k6, a WebSocket engine called ws, a built-in fake engine for practice, and in-house engines on its worker kit. Gatling and JMeter suites are imported and audited, but not run.

How do I install ai-perf?

It is not in the npx agentic-fx installer yet. Clone https://gitlab.com/agentic-fx/ai-perf, make sure you have Node 20 or later, and run node bin/perf.mjs doctor.

Sources

  1. ai-perf on GitLab
  2. ai-perf user guide
  3. ai-perf M7 evidence: measure honestly

Was this useful?

Only you and the owner see your answer and rating. Send private feedback

Enjoyed it? Support fx-blog with a coffee.

Several Machines, One p99 You Can Defend

Three storefront agents reported p99 of 43.52, 42.50 and 42.50 ms. Their average is 42.837; the fleet's p99 is 42.496. surge merges the agents' histograms bucket by bucket and evaluates the thresholds once, so several machines produce one verdict.

4 min readAdvanced

Comments

Oldest first

No comments yet. Start the conversation.

You sign in first; what you typed stays here. · 0 / 2,000