---
title: "A Performance Gate in CI That Doesn't Fail on Noise"
author: "Harry"
published: 2026-11-02T01:59:06.546Z
category: "Performance Testing"
tags: ["ai-perf"]
canonical: "https://fxblog.dev/posts/performance-testing-in-ci-gate"
---

# A Performance Gate in CI That Doesn't Fail on Noise

## Key takeaways

- The same build ran 22 % faster on a second run on the same laptop, more than a 10 % regression gate allows.
- perf ci exits 0 for PASS or WARN, 1 for FAIL, 2 for a refused spec and 3 for NOT EVALUABLE, and exit 3 should fail the job.
- A load run with no promoted baseline is NOT EVALUABLE, and only a person promotes a baseline, with --bootstrap and --promote.
- A regression needs p95 up more than 10 % and 5 ms, and a 95 % bootstrap lower bound above zero.
- perf trend flags a slow decay when a Theil–Sen line over recent runs moves more than 12 %, even when no single step crossed 10 %.

> **TL;DR** A perf gate in CI either fails merges on noise or lets unjudged builds through green. Performance testing in CI with ai-perf gates on the verdict: `perf ci` exits 3 when a build cannot be judged, and a regression must clear a 95 % confidence bound before it fails anything. Only here: a baseline measured under another workload, script, data or environment is STALE and is not compared at all.
>
> **Do this:** Add `perf ci <spec> --junit perf-junit.xml` to your merge-request pipeline and make exit code 3 fail the job.

[Part 2](02-capacity-testing-knee.md), **Where Does It Break? Find the Knee, Then the Cause**, ended with shop-api's full `db_pool` and a fix on the way. The next build runs the same staircase. Is it a regression, or a slow afternoon on the machine?

I ran one build twice on a real app, on the same laptop. The second run was 22 % faster on every operation. Nothing had changed but the machine's mood, and 22 % is more than the 10 % regression gate.

**This series, in three parts:**

1. **Your Load Test Passed. Did It Measure the Server?**
2. **Where Does It Break? Find the Knee, Then the Cause**
3. **A Performance Gate in CI That Doesn't Fail on Noise** (you are here)

## Gate merges on a smoke, and on its exit code

On every merge request, run a smoke: a few users, a short run, at most 10 users and 300 s. It proves the wiring and catches a broken build. A load profile belongs in a scheduled job on an environment its owner allows.

```yaml
  script:
    - node mock/server.mjs 9990 &
    # Wait until the mock answers: a target that is not up yet would refuse the run (exit 2).
    - for i in $(seq 1 40); do wget -qO- http://127.0.0.1:9990/health && break; sleep 0.5; done
    - node bin/perf.mjs ci .perf/demo/ci-smoke.json --root . --junit perf-junit.xml
```

| Exit | Means                                                                        |
| ---- | ---------------------------------------------------------------------------- |
| 0    | PASS or WARN                                                                 |
| 1    | FAIL, or an internal error                                                   |
| 2    | the spec was refused: production, a foreign host, a secret, a missing engine |
| 3    | NOT EVALUABLE                                                                |

> **Warning**
>
> Treat exit 3 as a failed job. An unjudged build is not a green build.

## Green numbers, red job

I ran `perf ci` on the repo's quickstart load spec, which had never been promoted:

```text
VERDICT: NOT_EVALUABLE
  NOT_EVALUABLE: no baseline at baselines/local/load.json (PS-F4); rerun with --bootstrap for the first run
```

Exit 3. All three transactions met their SLOs, and the JUnit file shows them passing beside one error, `run`. A load profile with nothing to compare against cannot answer "is this build slower?".

The way out is manual on purpose: run once with `--bootstrap`, read the result, then `perf report --bootstrap --promote`. CI never promotes on its own. Promote only a run you would sign.

## A regression has to clear the noise

A change counts only when p95 is up more than 10 % and more than 5 ms, and the 95 % lower bound of a bootstrap of both runs' histograms is above zero. A change past 10 % that is not significant at that sample size is a note, not a regression.

After my 22 % run, I changed one more thing. When the baseline is the same build at the same git sha under the same fingerprint, the verdict and `perf compare` say the difference is the environment's noise. The FAIL stays. The gate is not weakened.

> **Note**
>
> The fingerprint covers what moves the numbers: the workload, architecture, environment, script and data. Change one and the baseline is STALE. Nothing is compared until a new one is promoted.

## Trend catches what no single step shows

`perf trend` reads every comparable run and fits a Theil–Sen line over the last 8 since the last step, at least 5. One noisy run neither makes nor hides it.

| Repo check                                                 | Biggest single step | Flag         |
| ---------------------------------------------------------- | ------------------- | ------------ |
| five runs, latency planted at +6 % per run, +25 % in total | under 10 %          | `slow_decay` |
| four flat runs                                             | none                | nothing      |

A line moving more than 12 % the wrong way is the flag.

## When the run was ai-perf's fault

Some NOT EVALUABLE runs are the tool's own doing. Every judged run teaches a lesson, and a mistake in how the load was made is fixed in the next run of that profile: a saturated generator gets more workers. The run after checks the fix.

![The showcase's learning panel: attempt 1 NOT EVALUABLE with w0 at 97 % CPU, workers raised from 2 to 4, attempt 2 PASS with the fix verified, the spec file unchanged](https://fxblog.dev/media/01a199f7-7967-765f-821d-a61673d22d57/1600)

_The showcase's sample: the tool fixes its own generator, never your system or your SLOs_

It never touches the system under test, the planned load, the SLOs or the environment. A fix that recurs twice is rolled back and a person is asked.

## Try it

```sh
node bin/perf.mjs ci examples/quickstart/fake.json --root examples/quickstart --junit perf-junit.xml
echo $?
```

Then copy `ci/gitlab-ci.perf.yml` into your pipeline, point its smoke spec at your own service, and schedule the shop-api-style staircase from part 2 nightly against a promoted baseline.

## Read this with your AI

```text
Read the post "A Performance Gate in CI That Doesn't Fail on Noise".
1. Summarise it in 5 bullets for an engineer who owns a CI pipeline.
2. Ask me 3 questions to check I understood; wait, then correct me.
3. Look at my pipeline file and tell me where a perf smoke belongs and which exit codes should fail the job.
```

> **Success**
>
> You can gate merges on a perf verdict that fails on real regressions, not on noise or an empty baseline, which closes the series on shop-api's journey from a WARN to a pipeline.

## FAQ

### Should the load test run on every merge request?

No. A smoke of at most 10 users and 300 seconds runs on every merge request; a load or staircase profile belongs in a scheduled job on an authorised environment.

### What makes a baseline STALE?

A change to the workload, architecture, environment, script or data set. ai-perf then compares nothing until a new baseline is promoted.
