--- title: "RAG That Works: Why Retrieval Matters More Than the Prompt" author: "Harry" published: 2026-10-31T04:48:09.617Z updated: 2026-10-31T05:07:30.534Z category: "Building with AI" tags: ["prompts-guide"] canonical: "https://fxblog.dev/posts/rag-that-works-retrieval-first" --- # RAG That Works: Why Retrieval Matters More Than the Prompt ## Key takeaways - RAG answers from your documents by retrieving the few relevant chunks and grounding the answer in them. - The generation prompt needs three parts: answer only from the docs, an out when they lack the answer, and a quote for each claim. - Most bad RAG answers come from chunking and retrieval, not from the prompt. - Measure retrieval on its own by checking whether the right chunk reaches the top k, before tuning the prompt. - A large context window is a ceiling, not a target; most production systems combine RAG with a generous context. > **TL;DR** When a RAG answer is wrong, the prompt is the obvious suspect and usually the innocent one: most bad RAG answers come from chunking and retrieval. RAG finds the few chunks of your documents that answer a question and makes the model answer only from them. No prompt can use a chunk that never arrived, so measure whether the right chunk reached the top results before you touch the prompt. > > **Do this:** For each eval question, write down the chunk that holds the answer, run retrieval alone, and count how often that chunk lands in the top k. [Part 3](03-evals.md) left the test-case generator with cases that compile and pass, and a gap the eval cannot see: the generator cannot test a rule it has never read. The retry policy for webhook deliveries lives in your docs, not in the model. Ask the generator for tests of that policy without your docs, and it tests a policy it made up. Give it a grounded prompt and it answers "Not covered in the provided docs". Honest, and still wrong if the page is sitting in your docs folder. That second failure belongs to retrieval, and no prompt edit will fix it. **This series, in six parts:** 1. **Your Prompt Worked in Chat. Why Is the LLM API Different?**: the call, and what comes back. 2. **Build the Agent Loop Yourself, Once**: tools, and the loop every agent runs. 3. **Did Your Prompt Change Help? Prove It with an Eval**: a number for every change. 4. **RAG That Works: Why Retrieval Matters More Than the Prompt** (you are here): answers from your own docs. 5. **Your LLM Demo Works. What Every Call Needs in Production**: cost, failures, logs, injection. 6. **Your LLM Proxy Retries the Wrong Errors**: rate limits, retries and streams. ## Two phases: index once, retrieve per question ```text INGEST: docs → chunk (by structure, ~500-1500 tokens, with overlap) → embed each chunk → store in a vector DB (or even SQLite+numpy at small scale) QUERY: embed the question → top-k nearest chunks (+ optional keyword/BM25 hybrid and a rerank step) → build a grounded prompt → generate ``` An embedding turns text into a list of numbers so that similar meanings land close together. "Nearest chunks" are the ones closest in meaning to the question. At small scale you don't need a vector database: SQLite and numpy will do. ![RAG in two phases: at ingest, docs are chunked, embedded and stored; per question, the nearest chunks are retrieved, optionally reranked, and placed in a grounded prompt](https://fxblog.dev/media/01a19049-1cbd-7431-a898-aaae2550739c/1600) _Indexing happens once; retrieval happens on every question._ ## The prompt's whole job The generation prompt is short, and three parts of it carry the weight: ```text Answer using ONLY the documentation between tags. If the answer is not in the docs, say "Not covered in the provided docs" — do not use outside knowledge. Quote the relevant line for each claim. ...paste retrieved content... Question: What is the retry policy for webhook deliveries? ``` **Only** sets the scope. **An out** says what to do when the answer is missing, so the model stops instead of guessing. **Quoting** lets a person check each claim. Once those three are there, the prompt is rarely what failed. ## Measure retrieval on its own > **Warning** > > The model can only answer from what you put between the `` tags. If the right chunk was not retrieved, a perfect prompt still says "Not covered". So test retrieval without the model. It reuses the cases from part 3: 1. For each eval question, note which chunk holds the answer. 2. Run retrieval alone and count how often that chunk lands in the top k. 3. Fix chunking, add keyword (BM25) search, or add a rerank step, until that count is high. 4. Only then tune the generation prompt. For the webhook question, step 1 is one line in your eval file: the chunk id of the page that states the policy. When step 2 says that chunk was not in the top k, you know the prompt was never the problem, before you rewrite it. ## Doesn't a long context make RAG obsolete? Current Claude models take a million tokens. Why not paste the whole docs folder? | Approach | Good for | What it costs | | ------------------------------- | ------------------------------------- | ------------------------------------------------------------------------ | | Stuff everything in the context | One-off jobs over a few big documents | Every token, on every call; quality drops well before the window is full | | RAG | Many questions over a large corpus | Indexing once; small prompts per question | | Hybrid | Most production systems | RAG for scale, plus a generous context per question | > **Note** > > The window is a ceiling, not a target. The generator asks a question for every requirement it reads. Paying for the whole folder on each one is the expensive way to be less accurate. ## Try it ```text Here are three of my documents and five questions users ask: . Propose a chunking scheme (by which headings, how large, how much overlap), say for each question which chunk should be retrieved, and write the grounded generation prompt with an out and citations. ``` The generator now writes tests from your own docs, and you can tell a retrieval miss from a prompt miss. Next you want it running every night over every requirement. That brings the questions a demo never asks: what each run costs, what you log, and what happens when a retrieved doc says "ignore your instructions". Part 5 answers them. ## Read this with your AI ```text Read the post "RAG That Works: Why Retrieval Matters More Than the Prompt". 1. Summarise it in 5 bullets for a . 2. Ask me 3 questions to check I understood; wait, then correct me. 3. My RAG assistant gives this wrong answer: . Is it a retrieval or a generation problem, and what would you change first? ``` > **Success** > > You can tell a retrieval problem from a prompt problem. Next: **Your LLM Demo Works. What Every Call Needs in Production**. ## FAQ ### Do large context windows make RAG unnecessary? For one-off jobs over a few documents, stuffing the context can work. For many questions over a large corpus, RAG pays for indexing once instead of every token on every call, and most production systems combine both. ### How big should chunks be? Split by document structure, roughly 500 to 1,500 tokens, with some overlap, then adjust based on retrieval tests.