A Performance Gate in CI That Doesn't Fail on Noise
Performance testing in CI with ai-perf: a smoke on every merge request, the verdict as the exit code, regressions that must clear a 95 % confidence bound, STALE baselines, a trend for slow decay, and a learning loop for the load generator.
Key takeaways
- The same build ran 22 % faster on a second run on the same laptop, more than a 10 % regression gate allows.
- perf ci exits 0 for PASS or WARN, 1 for FAIL, 2 for a refused spec and 3 for NOT EVALUABLE, and exit 3 should fail the job.
- A load run with no promoted baseline is NOT EVALUABLE, and only a person promotes a baseline, with --bootstrap and --promote.
- A regression needs p95 up more than 10 % and 5 ms, and a 95 % bootstrap lower bound above zero.
- perf trend flags a slow decay when a Theil–Sen line over recent runs moves more than 12 %, even when no single step crossed 10 %.
Contents
FAQ
Should the load test run on every merge request?
No. A smoke of at most 10 users and 300 seconds runs on every merge request; a load or staircase profile belongs in a scheduled job on an authorised environment.
What makes a baseline STALE?
A change to the workload, architecture, environment, script or data set. ai-perf then compares nothing until a new baseline is promoted.
Sources
Only you and the owner see your answer and rating. Send private feedback
Enjoyed it? Support fx-blog with a coffee.
Related posts
Where Does It Break? Find the Knee, Then the Cause
Capacity testing with ai-perf: climb a staircase of steps, read the capacity from the steps that held, find the knee where latency breaks while throughput stops, and name the cause from signals such as a full connection pool.
Your Load Test Passed. Did It Measure the Server?
A load test can go green while it measures a saturated generator or a closed loop waiting on a frozen server. ai-perf runs the load in code, judges every run into one of four outcomes, and says NOT EVALUABLE when a run cannot be judged.
Several Machines, One p99 You Can Defend
Three storefront agents reported p99 of 43.52, 42.50 and 42.50 ms. Their average is 42.837; the fleet's p99 is 42.496. surge merges the agents' histograms bucket by bucket and evaluates the thresholds once, so several machines produce one verdict.
Comments
Oldest firstNo comments yet. Start the conversation.
Part 2, Where Does It Break? Find the Knee, Then the Cause, ended with shop-api's full
db_pooland a fix on the way. The next build runs the same staircase. Is it a regression, or a slow afternoon on the machine?I ran one build twice on a real app, on the same laptop. The second run was 22 % faster on every operation. Nothing had changed but the machine's mood, and 22 % is more than the 10 % regression gate.
This series, in three parts:
Gate merges on a smoke, and on its exit code
On every merge request, run a smoke: a few users, a short run, at most 10 users and 300 s. It proves the wiring and catches a broken build. A load profile belongs in a scheduled job on an environment its owner allows.
Green numbers, red job
I ran
perf cion the repo's quickstart load spec, which had never been promoted:Exit 3. All three transactions met their SLOs, and the JUnit file shows them passing beside one error,
run. A load profile with nothing to compare against cannot answer "is this build slower?".The way out is manual on purpose: run once with
--bootstrap, read the result, thenperf report --bootstrap --promote. CI never promotes on its own. Promote only a run you would sign.A regression has to clear the noise
A change counts only when p95 is up more than 10 % and more than 5 ms, and the 95 % lower bound of a bootstrap of both runs' histograms is above zero. A change past 10 % that is not significant at that sample size is a note, not a regression.
After my 22 % run, I changed one more thing. When the baseline is the same build at the same git sha under the same fingerprint, the verdict and
perf comparesay the difference is the environment's noise. The FAIL stays. The gate is not weakened.Trend catches what no single step shows
perf trendreads every comparable run and fits a Theil–Sen line over the last 8 since the last step, at least 5. One noisy run neither makes nor hides it.slow_decayA line moving more than 12 % the wrong way is the flag.
When the run was ai-perf's fault
Some NOT EVALUABLE runs are the tool's own doing. Every judged run teaches a lesson, and a mistake in how the load was made is fixed in the next run of that profile: a saturated generator gets more workers. The run after checks the fix.
It never touches the system under test, the planned load, the SLOs or the environment. A fix that recurs twice is rolled back and a person is asked.
Try it
Then copy
ci/gitlab-ci.perf.ymlinto your pipeline, point its smoke spec at your own service, and schedule the shop-api-style staircase from part 2 nightly against a promoted baseline.Read this with your AI