Your Prompt Worked in Chat. Why Is the LLM API Different?

TL;DR Move a chat prompt into code and you lose what the chat app did for you: the history, a long hidden system prompt, the format. Answers go flat, the model "forgets", and old sample code fails with HTTP 400. Over the LLM API you send everything on every call, write the system prompt yourself, set effort and a schema instead of asking in prose, and read the stop reason before the answer.

Do this: Find every place your code reads a model's answer and put a stop_reason check in front of it.

You paste the prompt that worked in the chat app into client.messages.create. The answer comes back flat and oddly formatted. Your follow-up gets a reply with no idea what you asked a minute ago. Then you copy a snippet from an older tutorial and get HTTP 400 for temperature.

None of that is a bug. The chat app was doing that work for you, and over the API it is yours.

This series, in six parts:

1. Your Prompt Worked in Chat. Why Is the LLM API Different? (you are here): the call, and what comes back.
2. Build the Agent Loop Yourself, Once: tools, and the loop every agent runs.
3. Did Your Prompt Change Help? Prove It with an Eval: a number for every change.
4. RAG That Works: Why Retrieval Matters More Than the Prompt: answers from your own docs.
5. Your LLM Demo Works. What Every Call Needs in Production: cost, failures, logs, injection.
6. Your LLM Proxy Retries the Wrong Errors: rate limits, retries and streams.

One feature runs through all six: a test-case generator that turns a requirement into test cases. Here it is a single call. Not covered: fine-tuning, and a parameter-by-parameter reference (the guide's Claude API page has that).

The API remembers nothing

Every request carries the whole conversation. If your code did not send an earlier turn, the model never saw it.

{
  "model": "...",
  "system": "You are ...",              // durable rules layer
  "messages": [
    {"role": "user", "content": "..."},
    {"role": "assistant", "content": "..."},
    {"role": "user", "content": "..."}   // newest turn
  ],
  "max_tokens": 16000
}

The system prompt is yours now

In the chat app, the vendor wrote a large hidden system prompt: tone, safety, formatting. Over the API you get a nearly raw model, which is why the same prompt reads flat. I'd spend more time on the system field than on the user message. It does the job a rules file does for an IDE agent.

Here is the test-case generator's first call:

import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY

response = client.messages.create(
    model="claude-opus-5-5",
    max_tokens=16000,
    system="You are a QA engineer. Answer with concrete, runnable examples.",
    messages=[{"role": "user", "content": "Design boundary tests for a field that accepts 1-100."}],
)

for block in response.content:
    if block.type == "text":       # content is a list of typed blocks — check type
        print(block.text)

The answer is a list of typed blocks, not a string. Check block.type before you read .text.

Settings replaced the prose

Three things people try to get with words are settings now:

With a schema, the generator hands back a TestCase your code can store. No regex:

from pydantic import BaseModel

class TestCase(BaseModel):
    id: str
    title: str
    steps: list[str]
    expected: str
    priority: str

response = client.messages.parse(
    model="claude-opus-5-5",
    max_tokens=16000,
    messages=[{"role": "user", "content": f"Create a test case for: {requirement}"}],
    output_format=TestCase,
)
case = response.parsed_output    # validated TestCase instance

Read the stop reason before the answer

A response that stopped early still looks like a response. stop_reason tells you which kind you got.

A refusal arrives with status 200, so your error handling never sees it. Only this field does.

Try it

Here is my API call: <paste code>.
Check it against current rules: model id, effort set explicitly,
no temperature/budget_tokens/prefill/forced tool_choice, stop_reason
checked, full history sent, stable content before volatile for
caching. List each problem with the fix.

The test-case generator now writes one case per call, but it cannot run a test, so it cannot check its own work against the orders suite. For that it needs a tool, and a loop around the tool. Part 2, Build the Agent Loop Yourself, writes that loop by hand.

Read this with your AI

Read the post "Your Prompt Worked in Chat. Why Is the LLM API Different?".
1. Summarise it in 5 bullets for a <your role>.
2. Ask me 3 questions to check I understood; wait, then correct me.
3. Here is a prompt that works for me in chat: <paste>.
   Write the system prompt it needs that the chat app was hiding.