Skip to content
fx-blog

Where Does It Break? Find the Knee, Then the Cause

Capacity testing with ai-perf: climb a staircase of steps, read the capacity from the steps that held, find the knee where latency breaks while throughput stops, and name the cause from signals such as a full connection pool.

Harry4 min readIntermediate
  • Intelligence
  • Verification
MarkdownSave

Key takeaways

  • A staircase such as 50, 100, 200 and 400 users holds each step long enough to settle and measure, which one ramp cannot do.
  • In ai-perf the capacity is the highest step of the unbroken run where every planned transaction met its SLA, and under 20 % headroom over the planned peak is a WARN.
  • A knee is where p95 more than doubles while throughput stops growing and the offered load just grew; rising latency at constant load is drift.
  • The analysis checks the load generator first, then the target's signals, then its CPU, and stops at the first answer.
  • A signal over its limit at the knee is named as the cause with its reading, and signals that stayed quiet are listed as ruled out.
Contents

TL;DR "It got slow at 400 users" is not a capacity and not a cause. Capacity testing in ai-perf climbs a staircase, reads the capacity from the steps that held, and finds the knee where latency breaks while throughput stops. Only here: the signal over its limit at the knee is named as the cause with its reading, and the quiet ones are listed as ruled out.

Do this: Plan your next stress test as steps, set the planned peak, and add one signal for your connection pool before the first loaded run.

Part 1, Your Load Test Passed. Did It Measure the Server?, left shop-api at WARN: it holds 200 users with 1.1× headroom. Where between 200 and 400 users did it break, and what was it waiting on? This part reads the same staircase and answers both.

The usual version of this goes: the load ramps to 400, a graph turns red somewhere, and someone spends the afternoon opening dashboards for the database, the app and the cache. At the end of it, "the database, probably" goes into the ticket.

This series, in three parts:

  1. Your Load Test Passed. Did It Measure the Server?
  2. Where Does It Break? Find the Knee, Then the Cause (you are here)
  3. A Performance Gate in CI That Doesn't Fail on Noise

Climb in steps, not one ramp

A ramp gives you one curve and no answer. A staircase holds each load long enough to measure it. The planning skill writes it like this:

jsonskills/planning-perf-tests/SKILL.md (stress or capacity)
"steps": [50, 100, 200, 400], "step_s": 300

Each step needs at least two windows: one to settle, one to measure. The capacity is the highest step of the unbroken run, from the bottom, where every planned transaction met its SLA. Below the planned peak is a FAIL. Under 20 % headroom is a WARN.

verdict.md for the sample shop-api staircase: 50, 100 and 200 users within the SLA, 400 users at p95 1840 ms outside it, WARN with 11 % headroom over a planned 180

That is shop-api's WARN: 200 users against a planned 180 is 11 %, under the 20 % line.

The knee is where latency breaks while throughput stops

ai-perf calls it a knee when p95 more than doubles over the early steady p95, rps stops growing, and the offered load just grew. In the sample: p95 96 → 1840 ms at t=600s, rps flat at 431.

The last condition came from a mistake. My first version found a knee in a soak test whose p95 doubled late, at constant load. That was drift, not saturation, so a knee now needs the load to have grown into that window or the one before.

From the knee to a cause

Once there is a knee, the analysis walks one fixed order and stops at the first answer:

mermaid
flowchart TD
  K[knee found] --> G{generator saturated?}
  G -- yes --> T[the numbers describe the tool]
  G -- no --> S{a signal over its limit?}
  S -- yes --> W[waiting on that signal, with its reading]
  S -- no --> C{target CPU above 85 %?}
  C -- yes --> B[CPU-bound]
  C -- no --> O{one core pegged?}
  O -- yes --> P[maybe CPU-bound on one thread]
  O -- no --> N[not CPU: a pool, locks, a dependency or I/O]

The generator comes first. If ai-perf's own workers were saturated, nothing after that is about your system.

Signals name the wait

CPU alone can only say "not CPU". To name the wait, /perf-env adds signals to env.json: one number each, read from the app's /metrics or from Prometheus.

jsondocs/user-guide.md §7 (env.json monitor.signals)
"signals": [
  { "name": "db_pool", "kind": "pool_used_pct", "prometheus": "http://prometheus:9090",
    "query": "100 * sum(hikaricp_connections_active) / sum(hikaricp_connections_max)" }
]

Two bottleneck sections for the same sample knee: without signals, "not CPU in api"; with signals, "waiting on db_pool", the pool at 100 % over 95 and locks never above 0

A signal that stayed under its limit is ruled out by name: here, locks. One that was over its limit all along, before the load grew, is mentioned but not blamed.

Try it

textPaste into Claude Code
/perf-plan
/perf-env

Ask for a stress profile with steps, then a db_pool signal for your pool. Now shop-api has a cause, and someone will raise the pool size. When the next build runs the same staircase, how do you tell a real regression from a laptop having a slow afternoon, without failing every other merge? Part 3, A Performance Gate in CI That Doesn't Fail on Noise, puts that question into the pipeline.

Read this with your AI

textPaste into your AI
Read the post "Where Does It Break? Find the Knee, Then the Cause".
1. Summarise it in 5 bullets for an engineer who runs stress tests.
2. Ask me 3 questions to check I understood; wait, then correct me.
3. Pick the metric in my stack that would show a full connection pool, and tell me its name.

FAQ

Where do signals come from?

Each signal is one number read during the run from the app's /metrics endpoint or from a Prometheus query, set in env.json under monitor.signals.

What if the target is not monitored at all?

The analysis says the resource is unknown. It can still find the knee from response times, but it will not guess a cause.

Sources

  1. ai-perf flows: from a knee to a cause
  2. ai-perf M8 evidence: test types
  3. ai-perf pipeline showcase

Was this useful?

Only you and the owner see your answer and rating. Send private feedback

Enjoyed it? Support fx-blog with a coffee.

Several Machines, One p99 You Can Defend

Three storefront agents reported p99 of 43.52, 42.50 and 42.50 ms. Their average is 42.837; the fleet's p99 is 42.496. surge merges the agents' histograms bucket by bucket and evaluates the thresholds once, so several machines produce one verdict.

4 min readAdvanced

Comments

Oldest first

No comments yet. Start the conversation.

You sign in first; what you typed stays here. · 0 / 2,000