Find the evidence, then answer from it

Some of the context lost in chunking can be put back at index time. And even a good search still produces invented answers if the writing instructions are loose.

Make each chunk explain itself

Contextual retrieval, published by Anthropic on 19 September 2024, attacks lost context when you store the chunks.

1. Split the document.
2. For each chunk, give a small model the full source and the chunk, and ask for a 50–100 token note on what the chunk is doing in that document.
3. Prepend the note and index the result for both meaning search and keyword search.

A chunk that only says "revenue rose 3% from the prior quarter" gets a prefix such as: "This is from ACME's Q2 2023 earnings release; prior-quarter revenue was $314 million." The chunk now matches "ACME Q2 revenue."

Two years on, this is closer to a tested default than to a novelty. If you already run search, it is the first upgrade to try.

• You pay once, at index time. Reusing a cached copy of the document, Anthropic estimated about $1.02 per million document tokens, assuming 800-token chunks, 8,000-token documents, and Claude 3 Haiku prices at publication.
• Pasting one document summary onto every chunk had limited effect. The point is a note written for that chunk.
• If a term is defined only in another document, put a glossary in the instruction that writes the note.
• The note is derived text. Store it apart from the source, and cite the source.

Each layer catches a different failure

Anthropic's figure is the share of questions where the right chunk is missing from the top 20.

No single technique fixes every miss. Context, keywords, and reranking each block a different failure. These numbers are Anthropic's test corpus. "Retrieval failures fell 67%" is not "answer accuracy rose 67%."

Fuse by rank, then spend money to narrow

Meaning scores and keyword scores are not on the same scale. Adding them lets one side dominate. The standard is reciprocal rank fusion: combine by rank, not by raw score.

A practical flow retrieves about 150 candidates so recall stays high, then reranks down to about 20. A reranker reads the question and the chunk together. It costs more than comparing precomputed numbers, and it is more accurate. Search wide and cheap; rerank narrow and expensive. In Anthropic's test, handing the model 20 chunks beat 5 or 10, but too many chunks distract the model. Measure it.

Fix the question without touching the index

About half of retrieval failures start in the question. These changes ship fast.

• Rewrite. Turn "how much was that?" into a question that stands alone.
• Multi-query. Search several phrasings and merge the lists.
• HyDE. Have the model draft a hypothetical answer, then search with that draft. The prose is often closer to the target document than the question was.
• Decompose. Split "compare A and B" into find-A and find-B.
• Filter first. Pull dates, departments, and document types out of the question and narrow the search before ranking.

Minimum rules for writing the answer

• Answer only from the provided material, and say you do not know when it is absent. That sentence alone does not remove hallucination, so pair it with a citation check.
• Attach a source chunk to each claim, and where you can, check afterward that the source actually supports the claim.
• Do not dump documents. Hand over an evidence package: the passage, document name, version, page, time range, relationships found, and places where sources disagree.
• Put the most relevant chunks at the start and the end, drop duplicates, and keep chunks from the same document in original order. Material buried in the middle is easy to miss.
• Extracting only the sentences that answer the question can cut tokens 30 to 60 percent. Compression can also drop facts, so evaluate it.
• Below a relevance threshold, say the material was not found and show what was searched. Do not treat the model's self-reported confidence as evidence.
• Separate facts that appear in the source from inferences drawn from them.