Skip to content
fx-blog

Closed Loops Hide the Cliff. Open Arrivals Find It.

Against one server frozen three seconds in every ten, 30 concurrent users measured p95 23 ms and 30 arrivals a second measured p95 2,523 ms. The closed loop backs off exactly when the system is in trouble, so its samples come from the windows where nothing was wrong.

Harry4 min readIntermediate
  • Judgment
  • Verification
MarkdownSave

Key takeaways

  • In a closed model the user count is fixed, so when the server stalls those users stop sending and the test stops sampling the slow window.
  • Against a server frozen 3 s in every 10 s, 30 concurrent users reported p95 23 ms while 30 arrivals a second reported p95 2,523 ms.
  • The closed run did record the stall, at max 3,020 ms and p99.9, but nowhere a threshold on p95 or p99 would have found it.
  • A percentile from a closed run is partly a measurement of the user count you chose; adding users changes it without the system changing.
  • Open load suits anything where traffic does not wait, such as a public API or a checkout; closed load suits sizing a fixed pool or client population.
  • surge switches between them with one field: model closed with vus, or model open with rate.
  • surge refuses a per_vu feeder under the open model, because each arrival is its own user and binding an identity to a pooled worker would be fiction.
Contents

TL;DR The open vs closed workload model decides whether your test can see a stall at all. Against one server frozen three seconds in every ten, 30 concurrent users measured p95 23 ms; 30 arrivals a second measured p95 2,523 ms. Same server, same freeze, 108× apart.

Do this: Find the load model in your own scenario. If it sets a number of users or threads, you are running closed, and your p95 is bounded by your own user count.

Part 1 left storefront red for a reason the thresholds named, and one doubt unnamed: the run was 30 users going round a loop. Here is what that shape cannot see.

A server that stops answering for three seconds in every ten has a problem. One of these two runs reported p95 23 ms against it, and nobody would have gone looking.

  1. Your Load Test Went Green. What Did It Measure?
  2. Closed Loops Hide the Cliff. Open Arrivals Find It.
  3. Write the Storefront Journey, Then Read the Verdict
  4. Several Machines, One p99 You Can Defend

Same server, same freeze, two answers

Both runs below hit fixtures/stall-target, which answers in 20 ms and freezes for 3 s in every 10. Both ran 40 seconds. Nothing else differs but the model.

closed · 30 users open · 30 arrivals/s
requests 37,890 1,199
p50 22 ms 22 ms
p95 23 ms 2,523 ms
p99 24 ms 2,916 ms
max 3,020 ms 3,017 ms

The medians agree. Everything a release decision rests on does not.

The open run's Over time tab: requests per second flat at 30, response time spiking to about 3,000 ms four times, and virtual users jumping from near zero to 69 at each spike

Look at the bottom chart. Virtual users sits near zero and jumps to 69 at each freeze: those are the arrivals that turned up while the server was not answering, all in flight at once. In the closed run that line is flat at 30 by definition, which is the whole difference in one picture.

Why the closed loop cannot see it

A closed model holds the number of users constant. Each user sends a request, waits for the answer, then sends the next one — so when the server stalls, those users stop sending.

The load backs off exactly when the system is in trouble. The test stops asking questions at the moment the answers would have been interesting.

mermaid
flowchart LR
  subgraph closed["closed: 30 users"]
    C1["server stalls"] --> C2["the 30 users block"]
    C2 --> C3["no new requests
    for 3 s"]
    C3 --> C4["samples come almost
    entirely from fast windows"]
  end
  subgraph open["open: 30 arrivals/s"]
    O1["server stalls"] --> O2["arrivals keep coming
    at 30/s"]
    O2 --> O3["~90 requests land
    inside each stall"]
    O3 --> O4["every one of them
    records seconds"]
  end

The freeze was in the closed numbers. It was just below p99.

The closed run did record the stall: its slowest request took 3,020 ms, and so did its 99.9th percentile. What it did not do was let the stall reach any number a threshold usually watches.

That is the trap. It is not an argument against closed load, only for knowing which question you asked.

Which to run

Closed answers "how does it behave with N users on it": the right question for a fixed client population or a pool you are sizing. Open answers "what happens to people arriving at R per second, whatever the system is doing": the right question for anything public.

Both are the same scenario with one field changed.

yamlfixtures/closed.yaml
load:
  model: closed
  vus: 30
  duration: 40s
yamlfixtures/open.yaml
load:
  model: open
  rate: 30
  duration: 40s

One rule surge enforces rather than leaving to you: a feeder with mode: per_vu is refused under the open model, because each arrival is its own user and binding an identity to whichever pooled worker picked it up would be fiction (scenario reference).

Try it

Try it
go run ./fixtures/stall-target -addr :18090 &
surge run fixtures/closed.yaml -json closed.json
surge run fixtures/open.yaml   -json open.json

Both of those runs printed PASS with a warning attached: no thresholds were set, so neither had any way to fail. A run that cannot fail is not a test, and the storefront scenario from part 1 is the shape that fixes it. In part 3 you write that scenario yourself, from an empty file to a verdict with three thresholds and an exit code CI can read.

Read this with your AI

textPaste into your AI
Read the post at https://fxblog.dev/posts/open-vs-closed-workload-model.
1. Summarise the difference between the two models in 5 bullets.
2. Ask me which model my current load test uses and why; wait, then correct me.
3. Rewrite my scenario in the other model and tell me what the new number would mean.

FAQ

Is the closed model wrong, then?

No. It answers a different question: how the system behaves with N users on it. That is the right question when the client population really is fixed, such as a connection pool or a kiosk fleet. It is the wrong question for a public endpoint, because real arrivals do not pause while the server recovers.

Why did the closed run make 37,890 requests and the open run only 1,199?

The closed run was never asked to hold a rate. Thirty users at about 22 ms each is roughly 1,360 requests a second in the seven good seconds and nothing in the other three, averaging 906 a second. The open run was asked for exactly 30 arrivals a second and delivered 29.97.

What happens in the open model when the system cannot keep up?

Concurrent iterations rise until they hit max_vus, which defaults to 1000. Arrivals beyond that are dropped and reported as dropped_iterations rather than quietly queued, so a run that could not deliver the rate you asked for says so.

Should I just raise the user count in a closed run instead?

It moves the number but does not fix the shape. More users still back off together during a stall, so you get a different p95 from the same server. If you want the percentile a visitor would see, ask for arrivals.

Sources

  1. surge — docs and downloads
  2. surge — scenario reference: load models and feeders
  3. surge — what it is
  4. The stall target and both scenarios used here

Was this useful?

Only you and the owner see your answer and rating. Send private feedback

Enjoyed it? Support fx-blog with a coffee.

Several Machines, One p99 You Can Defend

Three storefront agents reported p99 of 43.52, 42.50 and 42.50 ms. Their average is 42.837; the fleet's p99 is 42.496. surge merges the agents' histograms bucket by bucket and evaluates the thresholds once, so several machines produce one verdict.

4 min readAdvanced

Your Load Test Went Green. What Did It Measure?

A load test's number is always true about something — and it may be true about your laptop. surge watches its own CPU, marks a run untrustworthy above 85 % instead of reporting a pass, and keeps 512 bytes of what the server replied to every distinct failure.

4 min readBeginner

Comments

Oldest first

No comments yet. Start the conversation.

You sign in first; what you typed stays here. · 0 / 2,000