---
title: "Closed Loops Hide the Cliff. Open Arrivals Find It."
author: "Harry"
published: 2026-11-08T13:23:39.289Z
category: "Performance Testing"
tags: ["surge"]
canonical: "https://fxblog.dev/posts/open-vs-closed-workload-model"
---

# Closed Loops Hide the Cliff. Open Arrivals Find It.

## Key takeaways

- In a closed model the user count is fixed, so when the server stalls those users stop sending and the test stops sampling the slow window.
- Against a server frozen 3 s in every 10 s, 30 concurrent users reported p95 23 ms while 30 arrivals a second reported p95 2,523 ms.
- The closed run did record the stall, at max 3,020 ms and p99.9, but nowhere a threshold on p95 or p99 would have found it.
- A percentile from a closed run is partly a measurement of the user count you chose; adding users changes it without the system changing.
- Open load suits anything where traffic does not wait, such as a public API or a checkout; closed load suits sizing a fixed pool or client population.
- surge switches between them with one field: model closed with vus, or model open with rate.
- surge refuses a per\_vu feeder under the open model, because each arrival is its own user and binding an identity to a pooled worker would be fiction.

> **TL;DR** The open vs closed workload model decides whether your test can see a stall at all. Against one server frozen three seconds in every ten, 30 concurrent users measured p95 23 ms; 30 arrivals a second measured p95 2,523 ms. Same server, same freeze, 108× apart.
>
> **Do this:** Find the load model in your own scenario. If it sets a number of users or threads, you are running closed, and your p95 is bounded by your own user count.

[Part 1](https://fxblog.dev/posts/load-testing-tool-that-says-what-happened) left storefront red for a reason the thresholds named, and one doubt unnamed: the run was 30 users going round a loop. Here is what that shape cannot see.

A server that stops answering for three seconds in every ten has a problem. One of these two runs reported p95 23 ms against it, and nobody would have gone looking.

1. Your Load Test Went Green. What Did It Measure?
2. **Closed Loops Hide the Cliff. Open Arrivals Find It.**
3. Write the Storefront Journey, Then Read the Verdict
4. Several Machines, One p99 You Can Defend

[Closed Loops Hide the Cliff. Open Arrivals Find It.](https://www.youtube.com/watch?v=Pvo9NenK9-E)

## Same server, same freeze, two answers

Both runs below hit `fixtures/stall-target`, which answers in 20 ms and freezes for 3 s in every 10. Both ran 40 seconds. Nothing else differs but the model.

|          | closed · 30 users | open · 30 arrivals/s |
| -------- | ----------------- | -------------------- |
| requests | 37,890            | 1,199                |
| p50      | 22 ms             | 22 ms                |
| p95      | **23 ms**         | **2,523 ms**         |
| p99      | 24 ms             | 2,916 ms             |
| max      | 3,020 ms          | 3,017 ms             |

The medians agree. Everything a release decision rests on does not.

![The open run's Over time tab: requests per second flat at 30, response time spiking to about 3,000 ms four times, and virtual users jumping from near zero to 69 at each spike](https://fxblog.dev/media/01a1bb52-ba8b-7303-bcc1-a9f22e6cacf3/1600)

_Four freezes, four spikes. The arrival rate never bends around them — the concurrency does._

Look at the bottom chart. Virtual users sits near zero and jumps to 69 at each freeze: those are the arrivals that turned up while the server was not answering, all in flight at once. In the closed run that line is flat at 30 by definition, which is the whole difference in one picture.

## Why the closed loop cannot see it

A closed model holds the number of users constant. Each user sends a request, waits for the answer, then sends the next one — so when the server stalls, those users stop sending.

The load backs off exactly when the system is in trouble. The test stops asking questions at the moment the answers would have been interesting.

Where the samples come from

```mermaid
flowchart LR
  subgraph closed["closed: 30 users"]
    C1["server stalls"] --> C2["the 30 users block"]
    C2 --> C3["no new requests
    for 3 s"]
    C3 --> C4["samples come almost
    entirely from fast windows"]
  end
  subgraph open["open: 30 arrivals/s"]
    O1["server stalls"] --> O2["arrivals keep coming
    at 30/s"]
    O2 --> O3["~90 requests land
    inside each stall"]
    O3 --> O4["every one of them
    records seconds"]
  end
```

> **Note**
>
> This is why the closed run has 37,890 samples and the open run has 1,199. The closed loop was not asked to hold a rate; it went as fast as the server let it, which is 906 requests a second in the seven good seconds and nothing in the other three.

## The freeze was in the closed numbers. It was just below p99.

The closed run did record the stall: its slowest request took 3,020 ms, and so did its 99.9th percentile. What it did not do was let the stall reach any number a threshold usually watches.

> **Warning**
>
> A threshold on p95 or p99 against closed load is a threshold on how many users you chose. Add users and the same server reports a different p95.

That is the trap. It is not an argument against closed load, only for knowing which question you asked.

## Which to run

Closed answers "how does it behave with N users on it": the right question for a fixed client population or a pool you are sizing. Open answers "what happens to people arriving at R per second, whatever the system is doing": the right question for anything public.

Both are the same scenario with one field changed.

```yaml
load:
  model: closed
  vus: 30
  duration: 40s
```

```yaml
load:
  model: open
  rate: 30
  duration: 40s
```

> **Tip**
>
> Run the open model when you are deciding whether to ship. Run the closed model when you are sizing a pool. If you only ever run one, run open, because it is the one that can fail.

One rule surge enforces rather than leaving to you: a feeder with `mode: per_vu` is refused under the open model, because each arrival is its own user and binding an identity to whichever pooled worker picked it up would be fiction ([scenario reference](https://surge.fxblog.dev/docs/stable/scenario-reference)).

## Try it

```sh
go run ./fixtures/stall-target -addr :18090 &
surge run fixtures/closed.yaml -json closed.json
surge run fixtures/open.yaml   -json open.json
```

Both of those runs printed `PASS` with a warning attached: no thresholds were set, so neither had any way to fail. A run that cannot fail is not a test, and the storefront scenario from part 1 is the shape that fixes it. In part 3 you write that scenario yourself, from an empty file to a verdict with three thresholds and an exit code CI can read.

## Read this with your AI

```text
Read the post at https://fxblog.dev/posts/open-vs-closed-workload-model.
1. Summarise the difference between the two models in 5 bullets.
2. Ask me which model my current load test uses and why; wait, then correct me.
3. Rewrite my scenario in the other model and tell me what the new number would mean.
```

> **Success**
>
> You can now tell which question your load test asked, and why a p95 from a closed run is partly a measurement of your own user count. Next: **Write the Storefront Journey, Then Read the Verdict**.

## FAQ

### Is the closed model wrong, then?

No. It answers a different question: how the system behaves with N users on it. That is the right question when the client population really is fixed, such as a connection pool or a kiosk fleet. It is the wrong question for a public endpoint, because real arrivals do not pause while the server recovers.

### Why did the closed run make 37,890 requests and the open run only 1,199?

The closed run was never asked to hold a rate. Thirty users at about 22 ms each is roughly 1,360 requests a second in the seven good seconds and nothing in the other three, averaging 906 a second. The open run was asked for exactly 30 arrivals a second and delivered 29.97.

### What happens in the open model when the system cannot keep up?

Concurrent iterations rise until they hit max\_vus, which defaults to 1000. Arrivals beyond that are dropped and reported as dropped\_iterations rather than quietly queued, so a run that could not deliver the rate you asked for says so.

### Should I just raise the user count in a closed run instead?

It moves the number but does not fix the shape. More users still back off together during a stall, so you get a different p95 from the same server. If you want the percentile a visitor would see, ask for arrivals.
