Why retrieval-augmented generation fails so easily
Why a one-day RAG demo breaks in real work, how context gets lost, and the eight questions to answer before you pick an architecture.
On This Page
The tutorial version is simple. Collect documents, cut them to a fixed size, attach meaning coordinates, store them, fetch pieces that look like the question, and ask the model to write. A demo takes a day. In real work the answers drift.
The reason is also simple. This is not one technique. It is a pipeline of six steps, and the final quality is set by the weakest step.
What breaks at each step
| Step | What breaks | What the user sees |
|---|---|---|
| Parse | Tables, multi-column pages, scans, and charts fall apart as text | A number sticks to the wrong row |
| Chunk | The piece loses who, when, and which document | "Revenue rose 3%" with no company or quarter |
| Meaning search | Similar text is not the text that contains the answer. Names, codes, and numbers are weak spots | Asking for error TS-999 returns a nearby code |
| Ranking | A fixed cutoff drops the right piece just outside the list | The document has the answer, but the system says it does not |
| Generation | Retrieved evidence conflicts with what the model already "knows" | A number that disagrees with the citation, or a made-up claim |
| Operations | Changed or deleted documents stay searchable, and unauthorized documents mix in | An obsolete policy, or a leaked restricted file |
The underestimated failure: lost context
A chunk is the unit of search. Cut the boundary wrong and no later retrieval trick can recover the information. Context disappears in six ways.
| What disappears | Example | Result |
|---|---|---|
| What "this" points to | "this company", "that plan" | The subject is unknown |
| Document structure | Heading split from body, body split from appendix | The role of the passage is unknown |
| Table context | A row split from its column headers | The number attaches to the wrong item |
| Cause and effect | Cause in one chunk, effect in the next | The link is never found |
| Time | The old plan and the revised plan are separated | The stale plan is treated as current |
| Logical connection | Claim, evidence, and exception are separated | The answer is too sure, or simply wrong |
A contract is the clean example. Clause 4 says the customer pays a monthly amount. Clause 9 lists the refund exceptions. If "Can I get a refund?" retrieves only clause 4, the conclusion is wrong. Finding one chunk is not the same as finding enough evidence.
The test is short. If a person cannot understand the chunk by itself, the model cannot either.
Some questions do not fit this shape
Basic retrieval is strong when the answer sits in one place. Three kinds of questions are structurally weak.
- Global questions need a pass over the whole corpus, such as "what theme keeps recurring?" A handful of chunks cannot answer that.
- Multi-hop questions chain facts, such as "which customers did the owner of project A handle before?"
- Aggregation questions count or sum, such as "how many contracts last quarter?" That is a database query, not a search.
If you diagnose every failure as "the model is not good enough," the only remedy left is swapping models. Most failures are already decided while the material is being parsed, split, and stored.
Eight questions before you design
| Check | What to look at | What it does to the design |
|---|---|---|
| Size | Token count of the whole corpus | Under a few hundred thousand tokens, putting it all in context can be more accurate. Millions of tokens need a retrieval layer |
| Shape | Prose, tables, PDFs, slides, code, or chat | Tables want queries, visual documents want image search, code wants file navigation |
| Question mix | Single facts, summaries, connections, totals | Facts can stay on an improved basic stack. Connections and global questions need a graph or a compiled layer |
| Change rate | Write-once, weekly, or live | Frequent change punishes expensive rebuilds. Live data should be read from the source |
| Accuracy bar | Cost of a wrong answer, and whether citations are required | Higher bars add forced citations, reranking, and claim checks |
| Speed and cost | Seconds allowed, and budget per question | Live chat stays on one retrieval pass. Research work can search more than once |
| Access | Does each person see a different set of documents? | Filter permissions during search. Sensitive material may need a model you operate |
| Language | Korean only, or Korean mixed with English | You need morphological analysis and tests in each direction |
The first method follows the question.
| Question | Example | First method |
|---|---|---|
| Single fact | How many days is the refund window? | Hybrid search |
| Procedure or conditions | What are the five conditions for this exception? | Structure-aware chunks plus search |
| Compare documents | How do A and B differ in their supply chain? | Query expansion, or an agent |
| Relationship, multi-hop | How is the company contracted to X tied to lawsuit Y? | A graph |
| Corpus summary | What are the themes in 10,000 tickets? | Global graph search, or map-reduce summaries |
| Fresh or external | What changed in the policy yesterday? | Search plus a tool on the source system |
The first task is not a technology choice
Collect 50 to 100 questions real users will ask, and note where each answer lives. That is the evaluation set. With it, chunk size, search method, and reranking become numbers instead of taste.
A set of only easy questions lies. Include questions with no answer in the corpus, questions that require a connection, and questions about what changed recently.