Your Load Test Went Green. What Did It Measure?
A load test's number is always true about something — and it may be true about your laptop. surge watches its own CPU, marks a run untrustworthy above 85 % instead of reporting a pass, and keeps 512 bytes of what the server replied to every distinct failure.
Key takeaways
- A saturated load generator does not report an error; it reports a good number, because the load never arrived at the system under test.
- surge marks a run untrustworthy when its own machine passes 85 % CPU in the measured window, rather than reporting it as a pass.
- Failures are grouped by signature and each group keeps one real example with the first 512 bytes the server replied, so a million failures cost what fifty do.
- Request timings go into a log-linear histogram with about 3 % relative error in 15 KB of fixed memory, which is what makes merging across machines honest.
- Before quoting a load-test number, check whether the report says what the generator was doing while it measured.
- surge removed an object pool for request tracers because net/http's dial goroutines outlive their request and could write one request's timestamp into another's measurement.
Contents
FAQ
How do I know whether my load generator was the bottleneck?
Look for the generator's own CPU in the report for the measured window, not for the whole run. Above roughly 85 % the numbers describe the generator. surge records this and marks the run untrustworthy; if your tool does not record it, run the load from a second machine and see whether the result changes.
Why keep only 512 bytes of a failed response?
Enough to see which error it was, bounded so that a run with a million failures costs the memory of a few dozen. Failures are grouped by signature first, so you get one real example per distinct failure rather than a million copies of the same one.
Does surge need a cluster to be useful?
No. One process has been measured at 131,538 requests a second, which is past what most systems under test will absorb. Several machines are for load beyond one box or for generating from more than one location, and part 4 covers how the percentiles are merged.
Is this a replacement for monitoring the system under test?
No. surge measures what the client saw and attaches the evidence. What the server was doing while it answered that way comes from your own metrics; surge can push its timeline to Prometheus or InfluxDB so the two line up.
Sources
Only you and the owner see your answer and rating. Send private feedback
Enjoyed it? Support fx-blog with a coffee.
Related posts
Several Machines, One p99 You Can Defend
Three storefront agents reported p99 of 43.52, 42.50 and 42.50 ms. Their average is 42.837; the fleet's p99 is 42.496. surge merges the agents' histograms bucket by bucket and evaluates the thresholds once, so several machines produce one verdict.
Write the Storefront Journey, Then Read the Verdict
A load test with no thresholds has no way to fail, so its green means it finished. Write the thresholds first, then the journey: storefront's run came back FAIL, 2 of 3 breached, 979 of 3,604 requests failed — while p95 passed at 41.5 ms.
Closed Loops Hide the Cliff. Open Arrivals Find It.
Against one server frozen three seconds in every ten, 30 concurrent users measured p95 23 ms and 30 arrivals a second measured p95 2,523 ms. The closed loop backs off exactly when the system is in trouble, so its samples come from the windows where nothing was wrong.
Comments
Oldest firstNo comments yet. Start the conversation.
Last Friday a report said p95 41 ms and everyone relaxed. The laptop that produced it had been at 96 % CPU for the whole window, so 41 ms was how fast that laptop could ask, not how fast the system could answer. Nothing in the report said so.
Play on YouTube: Your Load Test Went Green. What Did It Measure?
The number is real. The question is whose limit it describes.
A load test produces a figure, and the figure is almost always true about something. The doubt is about which thing.
Three machines are in the picture: the one running the test, the network, and the system you care about. A percentile at the end cannot say which of them ran out first. When the generator gave way, the result looks like a system that is comfortably fast, because the load never arrived.
What a report has to carry for you to defend it
Ask these of any tool before you quote its number in a review:
Those four are the criteria surge was built to, and why its report is one self-contained HTML file. A number you have to defend six months later is a number you have to be able to re-read.
What surge does with them
Input is a scenario: a journey through your system, in YAML or in Go. Output is a verdict with its reasons.
In between it records every request into a log-linear histogram, about 3 % relative error in 15 KB of fixed memory (design); groups failures by signature and keeps 512 bytes of one real response per group; and watches its own CPU. Above 85 % the run is marked untrustworthy (install) instead of green.
One run of storefront, which did not go green
storefront is the journey this series follows: check the shop is up, read the catalogue, read it again with a stricter check, then an account endpoint that needs a token it never gets. It runs against the demo API in the surge repository with failure injection on, so these are measured failures, not described ones.
Three distinct failures out of 979, from 3,604 requests. The
×776is one bug, not 776 investigations.The bug that made me delete an optimisation
surge pooled its request tracers to save allocations. Go's
net/httpstarts goroutines to dial a connection, and those outlive the request that began them, so a tracer handed back to the pool could still take a timestamp from two requests ago and write it into whatever measurement was using it now.The race detector found the loud half. The quiet half would have shipped.
Try it
Binaries, the full flag list and the scenario reference are at surge.fxblog.dev.
storefront went red for a reason the thresholds named. But the run was 30 users going round a loop, and that shape has a blind spot: a server that freezes for three seconds in every ten can pass it. Part 2 runs the same journey closed and then open, against a target that stalls on purpose, and shows which one notices.
Read this with your AI