Your LLM Proxy Retries the Wrong Errors

TL;DR A proxy that retries the wrong errors wastes requests, fakes capacity, and can hand a client half an answer that looks whole. An LLM proxy makes one decision per failure: try elsewhere, wait, or return the error as it came. Limits belong to the organization, a 400 is never retried, and nothing is replayed after the first streamed byte.

Do this: Group your API keys by organization: every key in one organization is one rate limit, however many keys you hold.

Part 5 ended with the test-case generator and every other service sharing one rate limit, and a proxy that must decide, on every failure, whether another attempt can succeed.

Get it wrong three ways: A 400 retried three times returns the same error three times. Failover to a second key in the same organization hits the same 429. A retry after the first streamed byte hands the client two half-answers as one message.

This series, in six parts:

1. Your Prompt Worked in Chat. Why Is the LLM API Different?: the call, and what comes back.
2. Build the Agent Loop Yourself, Once: tools, and the loop every agent runs.
3. Did Your Prompt Change Help? Prove It with an Eval: a number for every change.
4. RAG That Works: Why Retrieval Matters More Than the Prompt: answers from your own docs.
5. Your LLM Demo Works. What Every Call Needs in Production: cost, failures, logs, injection.
6. Your LLM Proxy Retries the Wrong Errors (you are here): rate limits, retries and streams.

One question per failure

The decision deserves tests, so I wrote it as pure functions:

/** Whether a status is worth trying the next pool for. A 400 is the client's fault everywhere. */
export function isRetryable(status: number): boolean {
  return status === 429 || status === 401 || status === 403 || status >= 500;
}

A 401 or 403 is there because a different key may still work.

How long a backend rests depends on why:

I disable on a 401 rather than cool, because a revoked key never heals on a timer. And retry-after always beats my own backoff: the docs say earlier retries fail, so any wait I invent can only waste requests.

Rate limits belong to the organization

Anthropic enforces rate limits per organization. Two keys from one organization share one bucket, so failing over between them walks into the same 429. Keys in different organizations do not.

So the proxy tracks health per bucket, not per key: keys from one organization form one pool, and a 429 cools the pool. I pick the least recently used healthy pool.

Nothing is retried after the first byte

Before the first byte, a failure is invisible and the proxy can send the same body elsewhere. After it, the status is committed and the client has parsed message_start, the event that opens the stream. A replay pushes a second message_start into what the client thinks is one message, and the SDK either throws or glues two half-answers together.

So a mid-stream failure ends the stream with an error the client can see:

res.write(`event: error\ndata: ${errorEnvelope('api_error', `upstream failed mid-stream: ${error.message}`)}\n\n`);

1. Hold the response head until the upstream answers 2xx.
2. Pipe with backpressure, so a slow client slows the upstream read instead of filling memory.
3. Mid-stream, write the error event and end. Never close quietly.
4. Cool that backend, so the next request goes elsewhere.

Try it

Here is my proxy's retry code: <paste>. For each case (400, 401,
403, 429 with and without retry-after, 500, 529, network error,
failure after the first streamed byte), say what my code does and
what it should do. Then write one unit test per case.

The test-case generator went from one call to a loop, an eval, your own docs, a nightly run and a proxy that fails honestly. Build one piece this week? Make it the eval from part 3. Without it, every other change is a guess.

Read this with your AI

Read the post "Your LLM Proxy Retries the Wrong Errors".
1. Summarise it in 5 bullets for a <your role>.
2. Ask me 3 questions to check I understood; wait, then correct me.
3. My keys and organizations are below. Which of them share a rate
   limit, and does failover between them add any capacity?
<describe>