---
title: "Your LLM Proxy Retries the Wrong Errors"
author: "Harry"
published: 2026-10-31T04:48:10.377Z
updated: 2026-10-31T05:07:32.180Z
category: "Building with AI"
tags: ["prompts-guide"]
canonical: "https://fxblog.dev/posts/llm-proxy-retries-rate-limits-streaming"
---

# Your LLM Proxy Retries the Wrong Errors

## Key takeaways

- Anthropic enforces rate limits per organization, so failing over between two keys from the same organization adds no capacity.
- Track health per rate-limit bucket, not per key, and cool the bucket when it returns 429.
- Retry what another attempt could survive: 429, 5xx, network errors, and 401 or 403 on a different key; a 400 fails the same way everywhere.
- retry-after sets the wait exactly when present, and a 401 or 403 disables the key until a person fixes it.
- Once the first streamed byte reaches the client, a failure must end the stream with an error event, never a clean ending.
- The anthropic-ratelimit headers on every response, 429s included, give first-hand headroom for the current window at no extra cost.

> **TL;DR** A proxy that retries the wrong errors wastes requests, fakes capacity, and can hand a client half an answer that looks whole. An LLM proxy makes one decision per failure: try elsewhere, wait, or return the error as it came. Limits belong to the organization, a 400 is never retried, and nothing is replayed after the first streamed byte.
>
> **Do this:** Group your API keys by organization: every key in one organization is one rate limit, however many keys you hold.

[Part 5](05-production.md) ended with the test-case generator and every other service sharing one rate limit, and a proxy that must decide, on every failure, whether another attempt can succeed.

Get it wrong three ways: A 400 retried three times returns the same error three times. Failover to a second key in the same organization hits the same 429. A retry after the first streamed byte hands the client two half-answers as one message.

**This series, in six parts:**

1. **Your Prompt Worked in Chat. Why Is the LLM API Different?**: the call, and what comes back.
2. **Build the Agent Loop Yourself, Once**: tools, and the loop every agent runs.
3. **Did Your Prompt Change Help? Prove It with an Eval**: a number for every change.
4. **RAG That Works: Why Retrieval Matters More Than the Prompt**: answers from your own docs.
5. **Your LLM Demo Works. What Every Call Needs in Production**: cost, failures, logs, injection.
6. **Your LLM Proxy Retries the Wrong Errors** (you are here): rate limits, retries and streams.

## One question per failure

The decision deserves tests, so I wrote it as pure functions:

```ts
/** Whether a status is worth trying the next pool for. A 400 is the client's fault everywhere. */
export function isRetryable(status: number): boolean {
  return status === 429 || status === 401 || status === 403 || status >= 500;
}
```

A 401 or 403 is there because a _different_ key may still work.

![Each attempt picks a healthy bucket: 2xx streams back, 400 returns as-is, 429/5xx/network cools it and 401/403 disables its key before the next try; mid-stream, a failure ends the stream](https://fxblog.dev/media/01a19049-3e27-77b8-b5ba-f10b5f76a83b/1600)

_One question per failure: could another attempt succeed?_

How long a backend rests depends on why:

| Upstream says                      | Backend becomes | For how long                  |
| ---------------------------------- | --------------- | ----------------------------- |
| `429` with `retry-after: N`        | cooling         | exactly N seconds             |
| `429` spend cap (no `retry-after`) | cooling         | 15 minutes                    |
| `5xx`, network error               | cooling         | 2 s, 4 s, 8 s… capped at 60 s |
| `401`, `403`                       | disabled        | until a person fixes the key  |
| `400`, other 4xx                   | unchanged       | the error goes back as-is     |

I disable on a 401 rather than cool, because a revoked key never heals on a timer. And `retry-after` always beats my own backoff: the docs say earlier retries fail, so any wait I invent can only waste requests.

> **Note**
>
> Three attempts per request. If all fail, return the last upstream status and body as they came, not a 503 of your own: the client SDK reads that error to back off.

## Rate limits belong to the organization

Anthropic enforces rate limits per organization. Two keys from one organization share one bucket, so failing over between them walks into the same 429. Keys in different organizations do not.

So the proxy tracks health per bucket, not per key: keys from one organization form one pool, and a 429 cools the pool. I pick the least recently used healthy pool.

> **Warning**
>
> If all your keys are in one organization, the proxy is a monitor, not a multiplier. Spreading load across organizations to dodge a limit may break the provider's terms.

> **Tip**
>
> You can see a bucket's headroom for free. Every response, a 429 included, carries `anthropic-ratelimit-*` headers: requests and tokens remaining in the current window, and when it refills. Store them per backend, with the time you read them.

## Nothing is retried after the first byte

Before the first byte, a failure is invisible and the proxy can send the same body elsewhere. After it, the status is committed and the client has parsed `message_start`, the event that opens the stream. A replay pushes a second `message_start` into what the client thinks is one message, and the SDK either throws or glues two half-answers together.

So a mid-stream failure ends the stream with an error the client can see:

```ts
res.write(`event: error\ndata: ${errorEnvelope('api_error', `upstream failed mid-stream: ${error.message}`)}\n\n`);
```

1. Hold the response head until the upstream answers 2xx.
2. Pipe with backpressure, so a slow client slows the upstream read instead of filling memory.
3. Mid-stream, write the `error` event and end. Never close quietly.
4. Cool that backend, so the _next_ request goes elsewhere.

> **Danger**
>
> A truncated answer that ends cleanly looks complete. The client stores it, and the next turn builds on half an answer.

## Try it

```text
Here is my proxy's retry code: <paste>. For each case (400, 401,
403, 429 with and without retry-after, 500, 529, network error,
failure after the first streamed byte), say what my code does and
what it should do. Then write one unit test per case.
```

The test-case generator went from one call to a loop, an eval, your own docs, a nightly run and a proxy that fails honestly. Build one piece this week? Make it the eval from part 3. Without it, every other change is a guess.

## Read this with your AI

```text
Read the post "Your LLM Proxy Retries the Wrong Errors".
1. Summarise it in 5 bullets for a <your role>.
2. Ask me 3 questions to check I understood; wait, then correct me.
3. My keys and organizations are below. Which of them share a rate
   limit, and does failover between them add any capacity?
<describe>
```

> **Success**
>
> You can put a proxy in front of the model API that retries only what can succeed and never hides a failure. That's the series. Back to the start: **Your Prompt Worked in Chat. Why Is the LLM API Different?**

## FAQ

### Does rotating API keys get around a rate limit?

Not within one organization: its keys share one rate-limit bucket. Keys in separate organizations have separate limits, but using them to get around a limit may break the provider's terms of use.

### Should a proxy retry a 400 error?

No. A 400 means the request itself is invalid, so every backend rejects it the same way. Return it to the client unchanged.

### What should a proxy return when every backend fails?

The last upstream error as it came, with its status and body, rather than a generic 503 of its own, so the client SDK can decide how to back off.