RAG That Works: Why Retrieval Matters More Than the Prompt

TL;DR When a RAG answer is wrong, the prompt is the obvious suspect and usually the innocent one: most bad RAG answers come from chunking and retrieval. RAG finds the few chunks of your documents that answer a question and makes the model answer only from them. No prompt can use a chunk that never arrived, so measure whether the right chunk reached the top results before you touch the prompt.

Do this: For each eval question, write down the chunk that holds the answer, run retrieval alone, and count how often that chunk lands in the top k.

Part 3 left the test-case generator with cases that compile and pass, and a gap the eval cannot see: the generator cannot test a rule it has never read. The retry policy for webhook deliveries lives in your docs, not in the model.

Ask the generator for tests of that policy without your docs, and it tests a policy it made up. Give it a grounded prompt and it answers "Not covered in the provided docs". Honest, and still wrong if the page is sitting in your docs folder. That second failure belongs to retrieval, and no prompt edit will fix it.

This series, in six parts:

1. Your Prompt Worked in Chat. Why Is the LLM API Different?: the call, and what comes back.
2. Build the Agent Loop Yourself, Once: tools, and the loop every agent runs.
3. Did Your Prompt Change Help? Prove It with an Eval: a number for every change.
4. RAG That Works: Why Retrieval Matters More Than the Prompt (you are here): answers from your own docs.
5. Your LLM Demo Works. What Every Call Needs in Production: cost, failures, logs, injection.
6. Your LLM Proxy Retries the Wrong Errors: rate limits, retries and streams.

Two phases: index once, retrieve per question

INGEST:  docs → chunk (by structure, ~500-1500 tokens, with overlap)
         → embed each chunk → store in a vector DB (or even SQLite+numpy at small scale)
QUERY:   embed the question → top-k nearest chunks (+ optional keyword/BM25 hybrid
         and a rerank step) → build a grounded prompt → generate

An embedding turns text into a list of numbers so that similar meanings land close together. "Nearest chunks" are the ones closest in meaning to the question. At small scale you don't need a vector database: SQLite and numpy will do.

The prompt's whole job

The generation prompt is short, and three parts of it carry the weight:

Answer using ONLY the documentation between <docs> tags. If the answer
is not in the docs, say "Not covered in the provided docs" — do not use
outside knowledge. Quote the relevant line for each claim.

<docs>
...paste retrieved content...
</docs>

Question: What is the retry policy for webhook deliveries?

Only sets the scope. An out says what to do when the answer is missing, so the model stops instead of guessing. Quoting lets a person check each claim. Once those three are there, the prompt is rarely what failed.

Measure retrieval on its own

So test retrieval without the model. It reuses the cases from part 3:

1. For each eval question, note which chunk holds the answer.
2. Run retrieval alone and count how often that chunk lands in the top k.
3. Fix chunking, add keyword (BM25) search, or add a rerank step, until that count is high.
4. Only then tune the generation prompt.

For the webhook question, step 1 is one line in your eval file: the chunk id of the page that states the policy. When step 2 says that chunk was not in the top k, you know the prompt was never the problem, before you rewrite it.

Doesn't a long context make RAG obsolete?

Current Claude models take a million tokens. Why not paste the whole docs folder?

Try it

Here are three of my documents and five questions users ask: <paste>.
Propose a chunking scheme (by which headings, how large, how much
overlap), say for each question which chunk should be retrieved,
and write the grounded generation prompt with an out and citations.

The generator now writes tests from your own docs, and you can tell a retrieval miss from a prompt miss. Next you want it running every night over every requirement. That brings the questions a demo never asks: what each run costs, what you log, and what happens when a retrieved doc says "ignore your instructions". Part 5 answers them.

Read this with your AI

Read the post "RAG That Works: Why Retrieval Matters More Than the Prompt".
1. Summarise it in 5 bullets for a <your role>.
2. Ask me 3 questions to check I understood; wait, then correct me.
3. My RAG assistant gives this wrong answer: <question, answer,
   retrieved chunks>. Is it a retrieval or a generation problem,
   and what would you change first?