---
title: "Several Machines, One p99 You Can Defend"
author: "Harry"
published: 2026-11-08T13:23:40.373Z
category: "Performance Testing"
tags: ["surge"]
canonical: "https://fxblog.dev/posts/distributed-load-testing-one-p99"
---

# Several Machines, One p99 You Can Defend

## Key takeaways

- A percentile is a position in a sorted list, so averaging three machines' percentiles gives a number that is not a percentile of anything.
- Three storefront agents reported p99 43.52, 42.50 and 42.50 ms; their average is 42.837 ms and the merged p99 is 42.496 ms.
- surge merges the agents' log-linear histograms bucket by bucket and evaluates the thresholds once, so a fleet returns one verdict.
- Adding agents divides the scenario's load rather than multiplying it: 30 users over three agents was 10 peak VUs each.
- Every agent reports its own CPU, and a node above 85 % makes the run untrustworthy rather than a pass, exactly as on one machine.
- Read the per-node table before the percentile: an agent that contributed 10 % of the requests means you ran a smaller test than you think.
- surge -shard 3/5 divides load and feeder rows from the shard number alone, so machines that cannot reach each other still never share an identity.

> **TL;DR** In distributed load testing the average of three agents' p99 is not the fleet's p99. surge merges the agents' histograms bucket by bucket and evaluates the thresholds once, so three agents produce one verdict instead of three opinions.
>
> **Do this:** Look at how your tool combines runs from several machines. If it averages their percentiles, the number on your slide is not a percentile of anything.

[Part 3](https://fxblog.dev/posts/yaml-load-test-scenario) left storefront failing honestly from one machine, and asked what arrives with the second: if three agents each report a p99, which is the p99?

None of them. The three storefront agents reported 43.52, 42.496 and 42.496 ms. Their average is 42.837. The fleet's p99 is 42.496, and the gap is small only because all three watched one server from one machine.

1. Your Load Test Went Green. What Did It Measure?
2. Closed Loops Hide the Cliff. Open Arrivals Find It.
3. Write the Storefront Journey, Then Read the Verdict
4. **Several Machines, One p99 You Can Defend**

[Several Machines, One p99 You Can Defend](https://www.youtube.com/watch?v=pWSKQdd81Zs)

## Three agents, one verdict

The scenario does not change. The coordinator ships it to each agent, and their histograms come back.

```sh
surge agent -listen 0.0.0.0:7777 -token "$SURGE_TOKEN"

surge run fixtures/storefront.yaml \
  -agents gen1:7777,gen2:7777,gen3:7777 -html report.html
```

| node       | requests  | errors  | share  | peak VUs | p99          |
| ---------- | --------- | ------- | ------ | -------- | ------------ |
| gen-7771   | 1,207     | 329     | 33.4 % | 10       | 43.52 ms     |
| gen-7772   | 1,201     | 330     | 33.2 % | 10       | 42.50 ms     |
| gen-7773   | 1,206     | 332     | 33.4 % | 10       | 42.50 ms     |
| **merged** | **3,614** | **991** | —      | **30**   | **42.50 ms** |

> **Note**
>
> Peak VUs is 10 on each agent, not 30. The scenario asked for 30 users and the fleet divided them; adding agents did not multiply the load. To raise the load, raise the number in the scenario.

## Why an average of percentiles is not a percentile

A percentile is a position in a sorted list, and three sorted lists have no combined position you can reach by averaging three positions. Three class medians do not average into the median of the year.

surge records every request into a log-linear histogram: exact below 32 µs, then 32 sub-buckets per power of two, about 3 % error in 15 KB ([design](https://surge.fxblog.dev/docs/stable/design)). Merging two is adding their counts bucket by bucket; the percentile is read off the sum.

Merging, bucket by bucket

```mermaid
flowchart LR
  A["agent 1
  histogram"] --> M["add counts
  bucket by bucket"]
  B["agent 2
  histogram"] --> M
  C["agent 3
  histogram"] --> M
  M --> P["read p99 off
  the merged counts"]
  P --> T["thresholds
  evaluated once"]
```

> **Warning**
>
> In the run above the average was wrong by 0.34 ms, which would not have changed anyone's mind. Put one agent in another region and it is an average of three different distributions' tails, which is not the tail of anything.

## What the fleet report still has to tell you

Three machines do not make a number trustworthy. They make it bigger.

Every agent reports its own CPU, and part 1's rule applies per node: above 85 % the run is untrustworthy rather than a pass. Here each agent peaked at about 0.13 %, so no generator was the limit.

![The fleet report's Agents tab: three rows, each with 10 users, about 1,200 requests, a 33 per cent share, its own p95 and p99, a CPU peak of 0.1 per cent and a status of ok](https://fxblog.dev/media/01a1bb53-129e-74db-8213-a4664207b9d7/1600)

_The row that tells you whether any agent was the thing you measured. Share 33.4 / 33.2 / 33.4 per cent, CPU 0.1 per cent each._

> **Tip**
>
> Read the per-node table before the percentile. If one agent did 10 % of the requests, you ran a two-agent test and did not notice.

## Two ways to spread the load, and when to pick each

Agents are one way. The other needs no coordinator:

```sh
surge run api.yaml -shard 3/5 -json shard-3.json
surge merge shard-*.json -html report.html
```

`-shard 3/5` divides the load and the feeder rows from the shard number alone, so two machines never hand out the same account, and `surge merge` adds the raw histograms afterwards. Use it when the machines cannot reach each other. Use agents when you want one command and live progress.

## What this does not do

Several generators remove one limit and leave the rest.

| Still true with thirty agents                                | Why                                                                            |
| ------------------------------------------------------------ | ------------------------------------------------------------------------------ |
| The network path is part of the number                       | Three hosts in one rack do not measure what a distant user sees                |
| surge cannot say _why_ the server answered that way          | It measures what the client saw; the server's side comes from your own metrics |
| A scenario that asks the wrong question still gets an answer | Part 2's closed loop is a closed loop on thirty machines too                   |

Push the timeline to Prometheus or InfluxDB and both halves line up on one clock.

## Try it

```sh
surge agent -listen 127.0.0.1:7771 -node gen-a &
surge agent -listen 127.0.0.1:7772 -node gen-b &
surge run fixtures/storefront.yaml -agents 127.0.0.1:7771,127.0.0.1:7772 -html report.html
```

Two agents on one laptop is not a fleet, and the per-node table will say so. It is enough to watch one verdict come back from two histograms.

## Read this with your AI

```text
Read the post at https://fxblog.dev/posts/distributed-load-testing-one-p99.
1. Explain in 5 bullets why percentiles cannot be averaged across machines.
2. Ask me how my current tool combines multi-machine runs; wait, then tell me what that does to my p99.
3. Give me a 10-minute exercise to check it on my own last report.
```

> **Success**
>
> You can now run a load test from several machines and defend the single number that comes back — and say when it should not be defended. That is the series: a verdict with its reasons, from one machine or thirty. Scenarios, stall target, reports and docs are all at [surge.fxblog.dev](https://surge.fxblog.dev/) and the [repository](https://gitlab.com/agentic-fx/surge).

## FAQ

### How wrong is averaging percentiles in practice?

In the run in this post it was wrong by 0.34 ms, which would change nobody's decision, because all three agents watched the same server from one machine. The error grows with the difference between the agents: put one in another region and you are averaging the tails of three different distributions.

### Do I need agents, or will sharding do?

Sharding needs no coordinator and no network between the machines: each runs surge run -shard i/n and you merge the JSON afterwards. Use agents when you want one command, live progress, and the scenario and feeder files shipped to each node for you.

### Does adding agents increase the load?

No. The scenario states the load and the fleet divides it. Three agents running a 30-user scenario peak at 10 users each. To generate more load, raise the number in the scenario, then add agents if one machine cannot produce it.

### What does surge not tell me about a distributed run?

Why the server answered the way it did. It measures what each client saw and attaches the evidence. It also cannot tell you what distant users experience if every agent sits in one rack, because the network path from each agent is part of the measurement.
