Skip to content
fx-blog

Your Load Test Went Green. What Did It Measure?

A load test's number is always true about something — and it may be true about your laptop. surge watches its own CPU, marks a run untrustworthy above 85 % instead of reporting a pass, and keeps 512 bytes of what the server replied to every distinct failure.

Harry4 min readBeginner
  • Judgment
  • Verification
MarkdownSave

Key takeaways

  • A saturated load generator does not report an error; it reports a good number, because the load never arrived at the system under test.
  • surge marks a run untrustworthy when its own machine passes 85 % CPU in the measured window, rather than reporting it as a pass.
  • Failures are grouped by signature and each group keeps one real example with the first 512 bytes the server replied, so a million failures cost what fifty do.
  • Request timings go into a log-linear histogram with about 3 % relative error in 15 KB of fixed memory, which is what makes merging across machines honest.
  • Before quoting a load-test number, check whether the report says what the generator was doing while it measured.
  • surge removed an object pool for request tracers because net/http's dial goroutines outlive their request and could write one request's timestamp into another's measurement.
Contents

TL;DR A green load test can be a measurement of your own laptop. surge is a load testing tool that marks a run untrustworthy when its generator went over 85 % CPU, instead of printing the number anyway, and keeps the first 512 bytes the server replied to every distinct failure.

Do this: Open your last load-test report and look for the CPU of the machine that generated the load. If it isn't there, you cannot tell whose limit you measured.

Last Friday a report said p95 41 ms and everyone relaxed. The laptop that produced it had been at 96 % CPU for the whole window, so 41 ms was how fast that laptop could ask, not how fast the system could answer. Nothing in the report said so.

  1. Your Load Test Went Green. What Did It Measure?
  2. Closed Loops Hide the Cliff. Open Arrivals Find It.
  3. Write the Storefront Journey, Then Read the Verdict
  4. Several Machines, One p99 You Can Defend

The number is real. The question is whose limit it describes.

A load test produces a figure, and the figure is almost always true about something. The doubt is about which thing.

Three machines are in the picture: the one running the test, the network, and the system you care about. A percentile at the end cannot say which of them ran out first. When the generator gave way, the result looks like a system that is comfortably fast, because the load never arrived.

What a report has to carry for you to defend it

Ask these of any tool before you quote its number in a review:

  1. Does it watch its own machine, and say when it ran out?
  2. When a check fails, does it keep what the server actually replied?
  3. Is the percentile across several machines the real percentile, or an average of percentiles?
  4. Does the exit code mean something a CI job can act on?

Those four are the criteria surge was built to, and why its report is one self-contained HTML file. A number you have to defend six months later is a number you have to be able to re-read.

What surge does with them

Input is a scenario: a journey through your system, in YAML or in Go. Output is a verdict with its reasons.

In between it records every request into a log-linear histogram, about 3 % relative error in 15 KB of fixed memory (design); groups failures by signature and keeps 512 bytes of one real response per group; and watches its own CPU. Above 85 % the run is marked untrustworthy (install) instead of green.

One run of storefront, which did not go green

storefront is the journey this series follows: check the shop is up, read the catalogue, read it again with a stricter check, then an account endpoint that needs a token it never gets. It runs against the demo API in the surge repository with failure injection on, so these are measured failures, not described ones.

The Failures tab of a surge report, showing three distinct failures grouped by signature with counts of 776, 109 and 94, each with the URL, the check that failed and the first bytes the server replied

Three distinct failures out of 979, from 3,604 requests. The ×776 is one bug, not 776 investigations.

The bug that made me delete an optimisation

surge pooled its request tracers to save allocations. Go's net/http starts goroutines to dial a connection, and those outlive the request that began them, so a tracer handed back to the pool could still take a timestamp from two requests ago and write it into whatever measurement was using it now.

gointernal/engine/trace.go
// … Returning a tracer to a pool when the request function returns
// hands those still-live callbacks to whichever request draws the same
// tracer next, and there is no moment at which the last callback is
// known to have fired.
//
// The race detector found it as a write to the mutex itself … The
// quieter half is worse: a late ConnectStart from an abandoned dial
// writes its timestamp into another request's measurements, and a load
// tester whose numbers are occasionally another request's numbers has
// lost the only thing it is for.
func newTracer(start time.Time) *tracer {
	return &tracer{start: start}
}

The race detector found the loud half. The quiet half would have shipped.

Try it

Try it
go install gitlab.com/agentic-fx/surge/cmd/surge@latest
go run ./examples/target &
surge quick http://localhost:8080/health -vus 50 -duration 10s -threshold "p95 < 200ms"

Binaries, the full flag list and the scenario reference are at surge.fxblog.dev.

storefront went red for a reason the thresholds named. But the run was 30 users going round a loop, and that shape has a blind spot: a server that freezes for three seconds in every ten can pass it. Part 2 runs the same journey closed and then open, against a target that stalls on purpose, and shows which one notices.

Read this with your AI

textPaste into your AI
Read the post at https://fxblog.dev/posts/load-testing-tool-that-says-what-happened.
1. Summarise it in 5 bullets for a backend engineer who owns an API.
2. Ask me 3 questions to check I understood; wait, then correct me.
3. Look at my last load-test report and tell me which of the four criteria it fails.

FAQ

How do I know whether my load generator was the bottleneck?

Look for the generator's own CPU in the report for the measured window, not for the whole run. Above roughly 85 % the numbers describe the generator. surge records this and marks the run untrustworthy; if your tool does not record it, run the load from a second machine and see whether the result changes.

Why keep only 512 bytes of a failed response?

Enough to see which error it was, bounded so that a run with a million failures costs the memory of a few dozen. Failures are grouped by signature first, so you get one real example per distinct failure rather than a million copies of the same one.

Does surge need a cluster to be useful?

No. One process has been measured at 131,538 requests a second, which is past what most systems under test will absorb. Several machines are for load beyond one box or for generating from more than one location, and part 4 covers how the percentiles are merged.

Is this a replacement for monitoring the system under test?

No. surge measures what the client saw and attaches the evidence. What the server was doing while it answered that way comes from your own metrics; surge can push its timeline to Prometheus or InfluxDB so the two line up.

Sources

  1. surge — docs and downloads
  2. surge — what it is
  3. surge — installing, and what it needs from the machine
  4. surge — design and analysis
  5. surge — source

Was this useful?

Only you and the owner see your answer and rating. Send private feedback

Enjoyed it? Support fx-blog with a coffee.

Several Machines, One p99 You Can Defend

Three storefront agents reported p99 of 43.52, 42.50 and 42.50 ms. Their average is 42.837; the fleet's p99 is 42.496. surge merges the agents' histograms bucket by bucket and evaluates the thresholds once, so several machines produce one verdict.

4 min readAdvanced

Comments

Oldest first

No comments yet. Start the conversation.

You sign in first; what you typed stays here. · 0 / 2,000