Several Machines, One p99 You Can Defend
Three storefront agents reported p99 of 43.52, 42.50 and 42.50 ms. Their average is 42.837; the fleet's p99 is 42.496. surge merges the agents' histograms bucket by bucket and evaluates the thresholds once, so several machines produce one verdict.
Key takeaways
- A percentile is a position in a sorted list, so averaging three machines' percentiles gives a number that is not a percentile of anything.
- Three storefront agents reported p99 43.52, 42.50 and 42.50 ms; their average is 42.837 ms and the merged p99 is 42.496 ms.
- surge merges the agents' log-linear histograms bucket by bucket and evaluates the thresholds once, so a fleet returns one verdict.
- Adding agents divides the scenario's load rather than multiplying it: 30 users over three agents was 10 peak VUs each.
- Every agent reports its own CPU, and a node above 85 % makes the run untrustworthy rather than a pass, exactly as on one machine.
- Read the per-node table before the percentile: an agent that contributed 10 % of the requests means you ran a smaller test than you think.
- surge -shard 3/5 divides load and feeder rows from the shard number alone, so machines that cannot reach each other still never share an identity.
Contents
FAQ
How wrong is averaging percentiles in practice?
In the run in this post it was wrong by 0.34 ms, which would change nobody's decision, because all three agents watched the same server from one machine. The error grows with the difference between the agents: put one in another region and you are averaging the tails of three different distributions.
Do I need agents, or will sharding do?
Sharding needs no coordinator and no network between the machines: each runs surge run -shard i/n and you merge the JSON afterwards. Use agents when you want one command, live progress, and the scenario and feeder files shipped to each node for you.
Does adding agents increase the load?
No. The scenario states the load and the fleet divides it. Three agents running a 30-user scenario peak at 10 users each. To generate more load, raise the number in the scenario, then add agents if one machine cannot produce it.
What does surge not tell me about a distributed run?
Why the server answered the way it did. It measures what each client saw and attaches the evidence. It also cannot tell you what distant users experience if every agent sits in one rack, because the network path from each agent is part of the measurement.
Sources
Only you and the owner see your answer and rating. Send private feedback
Enjoyed it? Support fx-blog with a coffee.
Related posts
Write the Storefront Journey, Then Read the Verdict
A load test with no thresholds has no way to fail, so its green means it finished. Write the thresholds first, then the journey: storefront's run came back FAIL, 2 of 3 breached, 979 of 3,604 requests failed — while p95 passed at 41.5 ms.
Closed Loops Hide the Cliff. Open Arrivals Find It.
Against one server frozen three seconds in every ten, 30 concurrent users measured p95 23 ms and 30 arrivals a second measured p95 2,523 ms. The closed loop backs off exactly when the system is in trouble, so its samples come from the windows where nothing was wrong.
Your Load Test Went Green. What Did It Measure?
A load test's number is always true about something — and it may be true about your laptop. surge watches its own CPU, marks a run untrustworthy above 85 % instead of reporting a pass, and keeps 512 bytes of what the server replied to every distinct failure.
Comments
Oldest firstNo comments yet. Start the conversation.
Part 3 left storefront failing honestly from one machine, and asked what arrives with the second: if three agents each report a p99, which is the p99?
None of them. The three storefront agents reported 43.52, 42.496 and 42.496 ms. Their average is 42.837. The fleet's p99 is 42.496, and the gap is small only because all three watched one server from one machine.
Play on YouTube: Several Machines, One p99 You Can Defend
Three agents, one verdict
The scenario does not change. The coordinator ships it to each agent, and their histograms come back.
Why an average of percentiles is not a percentile
A percentile is a position in a sorted list, and three sorted lists have no combined position you can reach by averaging three positions. Three class medians do not average into the median of the year.
surge records every request into a log-linear histogram: exact below 32 µs, then 32 sub-buckets per power of two, about 3 % error in 15 KB (design). Merging two is adding their counts bucket by bucket; the percentile is read off the sum.
What the fleet report still has to tell you
Three machines do not make a number trustworthy. They make it bigger.
Every agent reports its own CPU, and part 1's rule applies per node: above 85 % the run is untrustworthy rather than a pass. Here each agent peaked at about 0.13 %, so no generator was the limit.
Two ways to spread the load, and when to pick each
Agents are one way. The other needs no coordinator:
-shard 3/5divides the load and the feeder rows from the shard number alone, so two machines never hand out the same account, andsurge mergeadds the raw histograms afterwards. Use it when the machines cannot reach each other. Use agents when you want one command and live progress.What this does not do
Several generators remove one limit and leave the rest.
Push the timeline to Prometheus or InfluxDB and both halves line up on one clock.
Try it
Two agents on one laptop is not a fleet, and the per-node table will say so. It is enough to watch one verdict come back from two histograms.
Read this with your AI