Skip to content
fx-blog

A Performance Gate in CI That Doesn't Fail on Noise

Performance testing in CI with ai-perf: a smoke on every merge request, the verdict as the exit code, regressions that must clear a 95 % confidence bound, STALE baselines, a trend for slow decay, and a learning loop for the load generator.

Harry4 min readIntermediate
  • Memory
  • Verification
MarkdownSave

Key takeaways

  • The same build ran 22 % faster on a second run on the same laptop, more than a 10 % regression gate allows.
  • perf ci exits 0 for PASS or WARN, 1 for FAIL, 2 for a refused spec and 3 for NOT EVALUABLE, and exit 3 should fail the job.
  • A load run with no promoted baseline is NOT EVALUABLE, and only a person promotes a baseline, with --bootstrap and --promote.
  • A regression needs p95 up more than 10 % and 5 ms, and a 95 % bootstrap lower bound above zero.
  • perf trend flags a slow decay when a Theil–Sen line over recent runs moves more than 12 %, even when no single step crossed 10 %.
Contents

TL;DR A perf gate in CI either fails merges on noise or lets unjudged builds through green. Performance testing in CI with ai-perf gates on the verdict: perf ci exits 3 when a build cannot be judged, and a regression must clear a 95 % confidence bound before it fails anything. Only here: a baseline measured under another workload, script, data or environment is STALE and is not compared at all.

Do this: Add perf ci <spec> --junit perf-junit.xml to your merge-request pipeline and make exit code 3 fail the job.

Part 2, Where Does It Break? Find the Knee, Then the Cause, ended with shop-api's full db_pool and a fix on the way. The next build runs the same staircase. Is it a regression, or a slow afternoon on the machine?

I ran one build twice on a real app, on the same laptop. The second run was 22 % faster on every operation. Nothing had changed but the machine's mood, and 22 % is more than the 10 % regression gate.

This series, in three parts:

  1. Your Load Test Passed. Did It Measure the Server?
  2. Where Does It Break? Find the Knee, Then the Cause
  3. A Performance Gate in CI That Doesn't Fail on Noise (you are here)

Gate merges on a smoke, and on its exit code

On every merge request, run a smoke: a few users, a short run, at most 10 users and 300 s. It proves the wiring and catches a broken build. A load profile belongs in a scheduled job on an environment its owner allows.

yamlci/gitlab-ci.perf.yml
  script:
    - node mock/server.mjs 9990 &
    # Wait until the mock answers: a target that is not up yet would refuse the run (exit 2).
    - for i in $(seq 1 40); do wget -qO- http://127.0.0.1:9990/health && break; sleep 0.5; done
    - node bin/perf.mjs ci .perf/demo/ci-smoke.json --root . --junit perf-junit.xml
Exit Means
0 PASS or WARN
1 FAIL, or an internal error
2 the spec was refused: production, a foreign host, a secret, a missing engine
3 NOT EVALUABLE

Green numbers, red job

I ran perf ci on the repo's quickstart load spec, which had never been promoted:

textnode bin/perf.mjs ci examples/quickstart/fake.json --root examples/quickstart --junit perf-junit.xml
VERDICT: NOT_EVALUABLE
  NOT_EVALUABLE: no baseline at baselines/local/load.json (PS-F4); rerun with --bootstrap for the first run

Exit 3. All three transactions met their SLOs, and the JUnit file shows them passing beside one error, run. A load profile with nothing to compare against cannot answer "is this build slower?".

The way out is manual on purpose: run once with --bootstrap, read the result, then perf report --bootstrap --promote. CI never promotes on its own. Promote only a run you would sign.

A regression has to clear the noise

A change counts only when p95 is up more than 10 % and more than 5 ms, and the 95 % lower bound of a bootstrap of both runs' histograms is above zero. A change past 10 % that is not significant at that sample size is a note, not a regression.

After my 22 % run, I changed one more thing. When the baseline is the same build at the same git sha under the same fingerprint, the verdict and perf compare say the difference is the environment's noise. The FAIL stays. The gate is not weakened.

Trend catches what no single step shows

perf trend reads every comparable run and fits a Theil–Sen line over the last 8 since the last step, at least 5. One noisy run neither makes nor hides it.

Repo check Biggest single step Flag
five runs, latency planted at +6 % per run, +25 % in total under 10 % slow_decay
four flat runs none nothing

A line moving more than 12 % the wrong way is the flag.

When the run was ai-perf's fault

Some NOT EVALUABLE runs are the tool's own doing. Every judged run teaches a lesson, and a mistake in how the load was made is fixed in the next run of that profile: a saturated generator gets more workers. The run after checks the fix.

The showcase's learning panel: attempt 1 NOT EVALUABLE with w0 at 97 % CPU, workers raised from 2 to 4, attempt 2 PASS with the fix verified, the spec file unchanged

It never touches the system under test, the planned load, the SLOs or the environment. A fix that recurs twice is rolled back and a person is asked.

Try it

Try it
node bin/perf.mjs ci examples/quickstart/fake.json --root examples/quickstart --junit perf-junit.xml
echo $?

Then copy ci/gitlab-ci.perf.yml into your pipeline, point its smoke spec at your own service, and schedule the shop-api-style staircase from part 2 nightly against a promoted baseline.

Read this with your AI

textPaste into your AI
Read the post "A Performance Gate in CI That Doesn't Fail on Noise".
1. Summarise it in 5 bullets for an engineer who owns a CI pipeline.
2. Ask me 3 questions to check I understood; wait, then correct me.
3. Look at my pipeline file and tell me where a perf smoke belongs and which exit codes should fail the job.

FAQ

Should the load test run on every merge request?

No. A smoke of at most 10 users and 300 seconds runs on every merge request; a load or staircase profile belongs in a scheduled job on an authorised environment.

What makes a baseline STALE?

A change to the workload, architecture, environment, script or data set. ai-perf then compares nothing until a new baseline is promoted.

Sources

  1. ai-perf sample GitLab CI job
  2. ai-perf M6 evidence: dogfood on a real app
  3. ai-perf M4 evidence: operations

Was this useful?

Only you and the owner see your answer and rating. Send private feedback

Enjoyed it? Support fx-blog with a coffee.

Several Machines, One p99 You Can Defend

Three storefront agents reported p99 of 43.52, 42.50 and 42.50 ms. Their average is 42.837; the fleet's p99 is 42.496. surge merges the agents' histograms bucket by bucket and evaluates the thresholds once, so several machines produce one verdict.

4 min readAdvanced

Comments

Oldest first

No comments yet. Start the conversation.

You sign in first; what you typed stays here. · 0 / 2,000