--- title: "Your Prompt Worked in Chat. Why Is the LLM API Different?" author: "Harry" published: 2026-10-31T04:48:08.222Z updated: 2026-10-31T05:07:28.639Z category: "Building with AI" tags: ["prompts-guide"] canonical: "https://fxblog.dev/posts/llm-api-basics-without-surprises" --- # Your Prompt Worked in Chat. Why Is the LLM API Different? ## Key takeaways - An LLM API is stateless, so every request must carry the whole conversation, and you pay for that history on every turn. - Over the API your system prompt does the job the chat app's hidden prompt used to do. - Effort, structured outputs and prompt caching replace instructions people used to write as prose. - Set effort explicitly: Opus 5.5 and Haiku 5.5 default to medium, most other models to high. - Read the stop reason on every response before the answer; a refusal arrives with HTTP 200. - temperature, top\_p, top\_k, budget\_tokens and assistant prefill return HTTP 400 on current Claude models. > **TL;DR** Move a chat prompt into code and you lose what the chat app did for you: the history, a long hidden system prompt, the format. Answers go flat, the model "forgets", and old sample code fails with HTTP 400. Over the LLM API you send everything on every call, write the system prompt yourself, set effort and a schema instead of asking in prose, and read the stop reason before the answer. > > **Do this:** Find every place your code reads a model's answer and put a `stop_reason` check in front of it. You paste the prompt that worked in the chat app into `client.messages.create`. The answer comes back flat and oddly formatted. Your follow-up gets a reply with no idea what you asked a minute ago. Then you copy a snippet from an older tutorial and get HTTP 400 for `temperature`. None of that is a bug. The chat app was doing that work for you, and over the API it is yours. **This series, in six parts:** 1. **Your Prompt Worked in Chat. Why Is the LLM API Different?** (you are here): the call, and what comes back. 2. **Build the Agent Loop Yourself, Once**: tools, and the loop every agent runs. 3. **Did Your Prompt Change Help? Prove It with an Eval**: a number for every change. 4. **RAG That Works: Why Retrieval Matters More Than the Prompt**: answers from your own docs. 5. **Your LLM Demo Works. What Every Call Needs in Production**: cost, failures, logs, injection. 6. **Your LLM Proxy Retries the Wrong Errors**: rate limits, retries and streams. One feature runs through all six: a test-case generator that turns a requirement into test cases. Here it is a single call. Not covered: fine-tuning, and a parameter-by-parameter reference (the guide's [Claude API page](https://gitlab.com/agentic-fx/ai-practices-guide/-/blob/main/03-api/claude-api.md) has that). ## The API remembers nothing Every request carries the whole conversation. If your code did not send an earlier turn, the model never saw it. ```jsonc { "model": "...", "system": "You are ...", // durable rules layer "messages": [ {"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}, {"role": "user", "content": "..."} // newest turn ], "max_tokens": 16000 } ``` > **Note** > > Stateless also means you pay for the history again on every turn. A 100K-token conversation that runs 40 more turns sends about 4M input tokens. ## The system prompt is yours now In the chat app, the vendor wrote a large hidden system prompt: tone, safety, formatting. Over the API you get a nearly raw model, which is why the same prompt reads flat. I'd spend more time on the `system` field than on the user message. It does the job a rules file does for an IDE agent. Here is the test-case generator's first call: ```python import anthropic client = anthropic.Anthropic() # reads ANTHROPIC_API_KEY response = client.messages.create( model="claude-opus-5-5", max_tokens=16000, system="You are a QA engineer. Answer with concrete, runnable examples.", messages=[{"role": "user", "content": "Design boundary tests for a field that accepts 1-100."}], ) for block in response.content: if block.type == "text": # content is a list of typed blocks — check type print(block.text) ``` The answer is a list of typed blocks, not a string. Check `block.type` before you read `.text`. ## Settings replaced the prose Three things people try to get with words are settings now: | You used to write | Now you set | | ------------------------------------ | -------------------------------------------------------------------------------------------- | | "Think step by step", "think harder" | **Effort**: `low`, `medium`, `high`, `xhigh`, `max` | | "Respond only with valid JSON" | **Structured outputs**: a schema the API guarantees | | The same long context on every call | **Prompt caching**: stable content first; cached reads cost about a tenth of the input price | With a schema, the generator hands back a `TestCase` your code can store. No regex: ```python from pydantic import BaseModel class TestCase(BaseModel): id: str title: str steps: list[str] expected: str priority: str response = client.messages.parse( model="claude-opus-5-5", max_tokens=16000, messages=[{"role": "user", "content": f"Create a test case for: {requirement}"}], output_format=TestCase, ) case = response.parsed_output # validated TestCase instance ``` > **Warning** > > **Set effort on every call.** Opus 5.5 and Haiku 5.5 default to `medium`; most other models default to `high`. Leave it out and a model upgrade changes how hard your app thinks, with no error. Part 3 comes back to this. ## Read the stop reason before the answer A response that stopped early still looks like a response. `stop_reason` tells you which kind you got. ![The request-response cycle over an LLM API: the request carries system prompt, history and settings; the response comes back with a stop reason that decides what your code does next](https://fxblog.dev/media/01a19048-f48b-7416-be05-e9a85bf0dc9f/1600) _Read the stop reason before you read the answer._ | `stop_reason` | Means | Do | | ------------- | ------------------------------------------- | --------------------------------------- | | `end_turn` | Finished | Use the answer | | `max_tokens` | Cut off mid-answer | Raise `max_tokens`, or stream | | `tool_use` | Wants a tool | Run it and loop (part 2) | | `refusal` | A safety classifier declined, with HTTP 200 | Handle it; read `stop_details.category` | A refusal arrives with status 200, so your error handling never sees it. Only this field does. > **Danger** > > Old sample code fails here with **HTTP 400** on current models: `temperature`, `top_p`, `top_k`, `thinking.budget_tokens` and an assistant-turn prefill. On Opus 5.5, Sonnet 5.5 and Fable 5.1, forced `tool_choice` (`any` or `tool`) fails too, and on Opus 5.5 and Sonnet 5.5 so does `thinking: {"type": "disabled"}`. Delete them. ## Try it ```text Here is my API call: . Check it against current rules: model id, effort set explicitly, no temperature/budget_tokens/prefill/forced tool_choice, stop_reason checked, full history sent, stable content before volatile for caching. List each problem with the fix. ``` The test-case generator now writes one case per call, but it cannot run a test, so it cannot check its own work against the orders suite. For that it needs a tool, and a loop around the tool. Part 2, Build the Agent Loop Yourself, writes that loop by hand. ## Read this with your AI ```text Read the post "Your Prompt Worked in Chat. Why Is the LLM API Different?". 1. Summarise it in 5 bullets for a . 2. Ask me 3 questions to check I understood; wait, then correct me. 3. Here is a prompt that works for me in chat: . Write the system prompt it needs that the chat app was hiding. ``` > **Success** > > You can make a correct call and read what comes back. Next: **Build the Agent Loop Yourself, Once**. ## FAQ ### Why does the model forget earlier messages over the API? The API keeps no memory between calls. Your code has to send the full message history on every request. ### Should I set effort explicitly? Yes. Default effort differs between models, so an upgrade can silently change how deeply your app reasons unless you set it.