Write the Storefront Journey, Then Read the Verdict
A load test with no thresholds has no way to fail, so its green means it finished. Write the thresholds first, then the journey: storefront's run came back FAIL, 2 of 3 breached, 979 of 3,604 requests failed — while p95 passed at 41.5 ms.
Key takeaways
- Write the thresholds before the journey; a scenario with no thresholds produces a pass that only means the run finished.
- check_rate is the threshold people leave out, and without it a run where every request returns 200 with the wrong body is green.
- storefront passed p95 at 41.5 ms and still failed: error_rate 27.16 % against a 1 % threshold, check_rate 80.32 % against 99 %.
- A ramp stage exists so you do not measure your connection pool filling up; starting 30 users at once measures the pool.
- Think time written as a range, 150ms..400ms, stops 30 users marching in step and producing a throughput sawtooth that is an artefact of the scenario.
- surge run exits non-zero when a threshold breaks, so a CI gate needs no output parsing.
- Read a report in this order: the verdict, which threshold broke and by how much, the warnings about whether the run was worth anything, then the failures.
Contents
FAQ
Which thresholds should I start with?
One latency percentile, one error rate and one check rate. The latency one is the number people quote, the error rate catches transport failures, and the check rate catches a server that answers 200 with the wrong body. storefront uses p95 < 150ms, error_rate < 1% and check_rate > 99%.
Why did p95 pass while the run failed?
Because latency was never the problem. The server answered quickly and answered wrongly: 27.16 % of requests failed and 1,073 checks did not hold. A gate that watches only latency would have called this run green.
Do I need the demo target to follow along?
No, but it makes the failures reproducible. examples/target in the surge repository serves the endpoints storefront uses and injects failures on demand with -fail, so the numbers in this post come out of a run you can repeat rather than out of a description.
Can I write the scenario in Go instead of YAML?
Yes. A scenario is a program, so anything you can call from Go can be measured: gRPC, SQL, a message queue, a vendor SDK. YAML covers HTTP and WebSocket, which is most of what people need; the Go library page covers the rest.
Sources
Only you and the owner see your answer and rating. Send private feedback
Enjoyed it? Support fx-blog with a coffee.
Related posts
Several Machines, One p99 You Can Defend
Three storefront agents reported p99 of 43.52, 42.50 and 42.50 ms. Their average is 42.837; the fleet's p99 is 42.496. surge merges the agents' histograms bucket by bucket and evaluates the thresholds once, so several machines produce one verdict.
Closed Loops Hide the Cliff. Open Arrivals Find It.
Against one server frozen three seconds in every ten, 30 concurrent users measured p95 23 ms and 30 arrivals a second measured p95 2,523 ms. The closed loop backs off exactly when the system is in trouble, so its samples come from the windows where nothing was wrong.
Your Load Test Went Green. What Did It Measure?
A load test's number is always true about something — and it may be true about your laptop. surge watches its own CPU, marks a run untrustworthy above 85 % instead of reporting a pass, and keeps 512 bytes of what the server replied to every distinct failure.
Comments
Oldest firstNo comments yet. Start the conversation.
Both runs in part 2 printed
PASSwith a warning attached: no thresholds were set, so neither had any way to fail. This is the file that fixes it.I write the thresholds first, before a single request. It feels backwards for about a minute, and then it stops you from writing a test that agrees with whatever happens.
Play on YouTube: Write the Storefront Journey, Then Read the Verdict
Start with what you are willing to promise
A threshold is the sentence you would say in a review. Write those three lines first and the rest of the file has something to be measured against.
Say how much load, and in what shape
storefront ramps to 30 users over five seconds and holds them for twenty-five. The ramp is not decoration: starting 30 users at once measures your connection pool filling up.
Part 2 is the decision hiding in that block. No
modelmeans closed, so these numbers are about 30 users, not about arrivals.Write the journey as the user walks it
Four steps, each with a name, a request, a check, and think time. The names are what you will read in the report, so name them after the user's intent.
Run it, and read four things
The console answers in the order you need it:
Read it in this order: the verdict, then which threshold broke and by how much, then the warnings about whether the run was worth anything, then the failures themselves.
The exit code is the point
surge runexits non-zero when a threshold breaks, so a CI job needs no output parsing:Keep
report.htmlas a build artefact. It is one self-contained file with the failures and their bodies in it, which is what you will want in three weeks when someone asks what happened.Try it
storefront now fails honestly, from one machine, at 30 users. The generator sat at 1 % CPU, so nothing here was limited by the laptop — but raise the load far enough and it will be, and then the number describes the laptop again, exactly as part 1 warned. Part 4 adds machines, and answers the question that comes with them: if three agents each report a p99, which of them is the p99?
Read this with your AI