Closed Loops Hide the Cliff. Open Arrivals Find It.
Against one server frozen three seconds in every ten, 30 concurrent users measured p95 23 ms and 30 arrivals a second measured p95 2,523 ms. The closed loop backs off exactly when the system is in trouble, so its samples come from the windows where nothing was wrong.
Key takeaways
- In a closed model the user count is fixed, so when the server stalls those users stop sending and the test stops sampling the slow window.
- Against a server frozen 3 s in every 10 s, 30 concurrent users reported p95 23 ms while 30 arrivals a second reported p95 2,523 ms.
- The closed run did record the stall, at max 3,020 ms and p99.9, but nowhere a threshold on p95 or p99 would have found it.
- A percentile from a closed run is partly a measurement of the user count you chose; adding users changes it without the system changing.
- Open load suits anything where traffic does not wait, such as a public API or a checkout; closed load suits sizing a fixed pool or client population.
- surge switches between them with one field: model closed with vus, or model open with rate.
- surge refuses a per_vu feeder under the open model, because each arrival is its own user and binding an identity to a pooled worker would be fiction.
Contents
FAQ
Is the closed model wrong, then?
No. It answers a different question: how the system behaves with N users on it. That is the right question when the client population really is fixed, such as a connection pool or a kiosk fleet. It is the wrong question for a public endpoint, because real arrivals do not pause while the server recovers.
Why did the closed run make 37,890 requests and the open run only 1,199?
The closed run was never asked to hold a rate. Thirty users at about 22 ms each is roughly 1,360 requests a second in the seven good seconds and nothing in the other three, averaging 906 a second. The open run was asked for exactly 30 arrivals a second and delivered 29.97.
What happens in the open model when the system cannot keep up?
Concurrent iterations rise until they hit max_vus, which defaults to 1000. Arrivals beyond that are dropped and reported as dropped_iterations rather than quietly queued, so a run that could not deliver the rate you asked for says so.
Should I just raise the user count in a closed run instead?
It moves the number but does not fix the shape. More users still back off together during a stall, so you get a different p95 from the same server. If you want the percentile a visitor would see, ask for arrivals.
Sources
Only you and the owner see your answer and rating. Send private feedback
Enjoyed it? Support fx-blog with a coffee.
Related posts
Several Machines, One p99 You Can Defend
Three storefront agents reported p99 of 43.52, 42.50 and 42.50 ms. Their average is 42.837; the fleet's p99 is 42.496. surge merges the agents' histograms bucket by bucket and evaluates the thresholds once, so several machines produce one verdict.
Write the Storefront Journey, Then Read the Verdict
A load test with no thresholds has no way to fail, so its green means it finished. Write the thresholds first, then the journey: storefront's run came back FAIL, 2 of 3 breached, 979 of 3,604 requests failed — while p95 passed at 41.5 ms.
Your Load Test Went Green. What Did It Measure?
A load test's number is always true about something — and it may be true about your laptop. surge watches its own CPU, marks a run untrustworthy above 85 % instead of reporting a pass, and keeps 512 bytes of what the server replied to every distinct failure.
Comments
Oldest firstNo comments yet. Start the conversation.
Part 1 left storefront red for a reason the thresholds named, and one doubt unnamed: the run was 30 users going round a loop. Here is what that shape cannot see.
A server that stops answering for three seconds in every ten has a problem. One of these two runs reported p95 23 ms against it, and nobody would have gone looking.
Play on YouTube: Closed Loops Hide the Cliff. Open Arrivals Find It.
Same server, same freeze, two answers
Both runs below hit
fixtures/stall-target, which answers in 20 ms and freezes for 3 s in every 10. Both ran 40 seconds. Nothing else differs but the model.The medians agree. Everything a release decision rests on does not.
Look at the bottom chart. Virtual users sits near zero and jumps to 69 at each freeze: those are the arrivals that turned up while the server was not answering, all in flight at once. In the closed run that line is flat at 30 by definition, which is the whole difference in one picture.
Why the closed loop cannot see it
A closed model holds the number of users constant. Each user sends a request, waits for the answer, then sends the next one — so when the server stalls, those users stop sending.
The load backs off exactly when the system is in trouble. The test stops asking questions at the moment the answers would have been interesting.
The freeze was in the closed numbers. It was just below p99.
The closed run did record the stall: its slowest request took 3,020 ms, and so did its 99.9th percentile. What it did not do was let the stall reach any number a threshold usually watches.
That is the trap. It is not an argument against closed load, only for knowing which question you asked.
Which to run
Closed answers "how does it behave with N users on it": the right question for a fixed client population or a pool you are sizing. Open answers "what happens to people arriving at R per second, whatever the system is doing": the right question for anything public.
Both are the same scenario with one field changed.
One rule surge enforces rather than leaving to you: a feeder with
mode: per_vuis refused under the open model, because each arrival is its own user and binding an identity to whichever pooled worker picked it up would be fiction (scenario reference).Try it
Both of those runs printed
PASSwith a warning attached: no thresholds were set, so neither had any way to fail. A run that cannot fail is not a test, and the storefront scenario from part 1 is the shape that fixes it. In part 3 you write that scenario yourself, from an empty file to a verdict with three thresholds and an exit code CI can read.Read this with your AI