Skip to content
fx-blog

Several Machines, One p99 You Can Defend

Three storefront agents reported p99 of 43.52, 42.50 and 42.50 ms. Their average is 42.837; the fleet's p99 is 42.496. surge merges the agents' histograms bucket by bucket and evaluates the thresholds once, so several machines produce one verdict.

Harry4 min readAdvanced
  • Judgment
  • Verification
MarkdownSave

Key takeaways

  • A percentile is a position in a sorted list, so averaging three machines' percentiles gives a number that is not a percentile of anything.
  • Three storefront agents reported p99 43.52, 42.50 and 42.50 ms; their average is 42.837 ms and the merged p99 is 42.496 ms.
  • surge merges the agents' log-linear histograms bucket by bucket and evaluates the thresholds once, so a fleet returns one verdict.
  • Adding agents divides the scenario's load rather than multiplying it: 30 users over three agents was 10 peak VUs each.
  • Every agent reports its own CPU, and a node above 85 % makes the run untrustworthy rather than a pass, exactly as on one machine.
  • Read the per-node table before the percentile: an agent that contributed 10 % of the requests means you ran a smaller test than you think.
  • surge -shard 3/5 divides load and feeder rows from the shard number alone, so machines that cannot reach each other still never share an identity.
Contents

TL;DR In distributed load testing the average of three agents' p99 is not the fleet's p99. surge merges the agents' histograms bucket by bucket and evaluates the thresholds once, so three agents produce one verdict instead of three opinions.

Do this: Look at how your tool combines runs from several machines. If it averages their percentiles, the number on your slide is not a percentile of anything.

Part 3 left storefront failing honestly from one machine, and asked what arrives with the second: if three agents each report a p99, which is the p99?

None of them. The three storefront agents reported 43.52, 42.496 and 42.496 ms. Their average is 42.837. The fleet's p99 is 42.496, and the gap is small only because all three watched one server from one machine.

  1. Your Load Test Went Green. What Did It Measure?
  2. Closed Loops Hide the Cliff. Open Arrivals Find It.
  3. Write the Storefront Journey, Then Read the Verdict
  4. Several Machines, One p99 You Can Defend

Three agents, one verdict

The scenario does not change. The coordinator ships it to each agent, and their histograms come back.

sheach generator, then the coordinator
surge agent -listen 0.0.0.0:7777 -token "$SURGE_TOKEN"

surge run fixtures/storefront.yaml \
  -agents gen1:7777,gen2:7777,gen3:7777 -html report.html
node requests errors share peak VUs p99
gen-7771 1,207 329 33.4 % 10 43.52 ms
gen-7772 1,201 330 33.2 % 10 42.50 ms
gen-7773 1,206 332 33.4 % 10 42.50 ms
merged 3,614 991 — 30 42.50 ms

Why an average of percentiles is not a percentile

A percentile is a position in a sorted list, and three sorted lists have no combined position you can reach by averaging three positions. Three class medians do not average into the median of the year.

surge records every request into a log-linear histogram: exact below 32 µs, then 32 sub-buckets per power of two, about 3 % error in 15 KB (design). Merging two is adding their counts bucket by bucket; the percentile is read off the sum.

mermaid
flowchart LR
  A["agent 1
  histogram"] --> M["add counts
  bucket by bucket"]
  B["agent 2
  histogram"] --> M
  C["agent 3
  histogram"] --> M
  M --> P["read p99 off
  the merged counts"]
  P --> T["thresholds
  evaluated once"]

What the fleet report still has to tell you

Three machines do not make a number trustworthy. They make it bigger.

Every agent reports its own CPU, and part 1's rule applies per node: above 85 % the run is untrustworthy rather than a pass. Here each agent peaked at about 0.13 %, so no generator was the limit.

The fleet report's Agents tab: three rows, each with 10 users, about 1,200 requests, a 33 per cent share, its own p95 and p99, a CPU peak of 0.1 per cent and a status of ok

Two ways to spread the load, and when to pick each

Agents are one way. The other needs no coordinator:

shon each of five machines
surge run api.yaml -shard 3/5 -json shard-3.json
surge merge shard-*.json -html report.html

-shard 3/5 divides the load and the feeder rows from the shard number alone, so two machines never hand out the same account, and surge merge adds the raw histograms afterwards. Use it when the machines cannot reach each other. Use agents when you want one command and live progress.

What this does not do

Several generators remove one limit and leave the rest.

Still true with thirty agents Why
The network path is part of the number Three hosts in one rack do not measure what a distant user sees
surge cannot say why the server answered that way It measures what the client saw; the server's side comes from your own metrics
A scenario that asks the wrong question still gets an answer Part 2's closed loop is a closed loop on thirty machines too

Push the timeline to Prometheus or InfluxDB and both halves line up on one clock.

Try it

Try it
surge agent -listen 127.0.0.1:7771 -node gen-a &
surge agent -listen 127.0.0.1:7772 -node gen-b &
surge run fixtures/storefront.yaml -agents 127.0.0.1:7771,127.0.0.1:7772 -html report.html

Two agents on one laptop is not a fleet, and the per-node table will say so. It is enough to watch one verdict come back from two histograms.

Read this with your AI

textPaste into your AI
Read the post at https://fxblog.dev/posts/distributed-load-testing-one-p99.
1. Explain in 5 bullets why percentiles cannot be averaged across machines.
2. Ask me how my current tool combines multi-machine runs; wait, then tell me what that does to my p99.
3. Give me a 10-minute exercise to check it on my own last report.

FAQ

How wrong is averaging percentiles in practice?

In the run in this post it was wrong by 0.34 ms, which would change nobody's decision, because all three agents watched the same server from one machine. The error grows with the difference between the agents: put one in another region and you are averaging the tails of three different distributions.

Do I need agents, or will sharding do?

Sharding needs no coordinator and no network between the machines: each runs surge run -shard i/n and you merge the JSON afterwards. Use agents when you want one command, live progress, and the scenario and feeder files shipped to each node for you.

Does adding agents increase the load?

No. The scenario states the load and the fleet divides it. Three agents running a 30-user scenario peak at 10 users each. To generate more load, raise the number in the scenario, then add agents if one machine cannot produce it.

What does surge not tell me about a distributed run?

Why the server answered the way it did. It measures what each client saw and attaches the evidence. It also cannot tell you what distant users experience if every agent sits in one rack, because the network path from each agent is part of the measurement.

Sources

  1. surge — docs and downloads
  2. surge — across several machines
  3. surge — design and analysis: the histogram
  4. surge — source

Was this useful?

Only you and the owner see your answer and rating. Send private feedback

Enjoyed it? Support fx-blog with a coffee.

Your Load Test Went Green. What Did It Measure?

A load test's number is always true about something — and it may be true about your laptop. surge watches its own CPU, marks a run untrustworthy above 85 % instead of reporting a pass, and keeps 512 bytes of what the server replied to every distinct failure.

4 min readBeginner

Comments

Oldest first

No comments yet. Start the conversation.

You sign in first; what you typed stays here. · 0 / 2,000