Measure it, run it, and avoid the usual mistakes

If you cannot measure it, you cannot improve it. Evaluation has two layers. Mix them and you cannot tell what broke. The first split is whether retrieval found the evidence and the answer was still wrong, or retrieval never found it.

What to measure

Run six setups on the same corpus and the same questions. You do not need the full archive. 100 to 200 representative questions are enough.

Include single facts, exact identifiers such as product numbers, relationships across documents, corpus-level trends, numeric totals, contradictory documents, and questions with no answer in the material.

Before you add a more complex structure, ask how much more accurate it is than plain hybrid search, which question types improved, what the extra storage and update cost is, how much slower and more expensive each question became, and whether you can operate the extra complexity. The best architecture is the one with the best balance of accuracy and operating cost, not the most elaborate one.

Operating rules:

• Build an evaluation set of at least 50 to 100 real user questions before you go live.
• If you change chunking, the embedding model, how many chunks you fetch, or the instructions, rerun the set. A change without a measurement is a dice roll.
• Keep a path for failed questions. "Not found" logs and user reactions become the next evaluation set and the priority list.
• Do not treat a published benchmark as your system's number. It tells you which layer fixes which failure. The size of the effect is measured on your data.

What separates a demo from a service

• Delete everywhere. If a removed document remains in search, the vector index, the graph, the wiki, or the cache, you will answer with a retired rule.
• Versions and valid time distinguish current from past. Relationships on a graph need the same.
• Permissions belong in retrieval, as a filter on the search. Censoring after the answer is written is already late.
• Provenance lets you trace a claim, a relationship, or a summary back to a source passage.
• Cost splits into index time (embeddings and context notes, paid once) and question time (search, rerank, and answer tokens, paid every time). Prompt caching is the largest saving on both sides. If question cost is the problem, reduce the candidate count and remeasure.
• Rerank candidate count and agent loop count dominate latency. Set a time limit per stage and fall back when it is exceeded.
• Log the question, the rewritten question, the chunk ids, the rerank scores, and the final citations. Without that log you can never answer "why did it say that?"
• Law, medicine, and finance need a person in the review path for high-stakes answers.

A retrieved document can contain "ignore the previous instructions." If the model treats that as a command, a hostile document steers the system. OWASP's retrieval security guidance points out new attack surface from collection through storage, search, answering, and tool use. Retrieved text is always data, never an instruction to execute. That matters most when you ingest the public web or email. Showing one user's documents to another user is the same class of failure.

For Korean operations, particles and endings make keyword precision unstable, so check the morphological analyzer, and test the embedding model on Korean and on mixed Korean-English questions from your own files.

Build from the simplest baseline

Stage 0, diagnose. Convert the corpus to tokens. Collect 50 to 100 questions and where the answers live. Note the mix of single-place, global, totals, and live questions. You do not proceed without an evaluation set.

Stage 1, evidence. Parsing, immutable originals, labels, hybrid search, basic citations. If you are under a few hundred thousand tokens, also try putting it all in context with a cache. Done means you can point at a source location and cite it. Split failures into "did not find" and "found but answered wrong."

Stage 2, context. Try structure-aware splits, parent-child, contextual retrieval, and reranking one at a time, and remeasure each time. Many retrieval misses point to contextual retrieval and reranking. Many answer misses point to forced citations and a claim check. If the question is the problem, rewriting, multi-query, and HyDE are the cheapest fixes because they do not touch the index.

Stage 3, route. Totals go to SQL, global and connection questions to a graph or wiki, live data to MCP. Extract the entities and relationships that actually recur. Start with the highest-value slice, not a full-corpus graph. Done means relationship questions and trend questions measurably improve.

Stage 4, coordinate. Add a router, agentic search, business tools, automatic checks, and incremental updates. Write down the retry cap, the time limit, and the rule for stopping when evidence is thin.

Stage 5, operate. Fold failed questions back into the evaluation set and repeat. When a wiki outgrows its index, add local hybrid search such as qmd. When different people need different permissions, move to the stage-2 retrieval setup.

Design checklist

Material and splits. Does the parser keep tables, headings, and lists? Can a person understand a chunk alone? Do document types use different split rules? Does every chunk have source, date, and permission? Is there a rule for old versions and duplicates? Are the original, the extracted text, the table, and the location linked?

Store and find. Do you have both meaning search and keyword search? Did you consider a context note on each chunk? Do you fuse the two lists by rank (RRF)? Are the rerank stage and the final count decided? Are permissions applied during search? Is there at least query rewrite or multi-query? Did you test Korean morphology and mixed-language questions?

Answer and trust. Are citations required, and is citation-to-claim checked? Is there a path that says "not found"? Are placement and deduplication defined? Are facts separated from inferences? Are instructions inside documents treated only as data?

Evaluation and operations. Does the evaluation set exist before launch? Do you remeasure when the setup changes? Do you collect failures? Does deletion propagate to every store? Are version and valid time recorded? Are questions and citations logged? Does a person review high-stakes answers?

Ten things not to do

1. Tuning chunk size before you have an evaluation set. Optimization without measurement is superstition.
2. Sending every question to meaning search alone. Codes, names, and dates will fail. Hybrid is the default.
3. Teaching facts with fine-tuning. The knowledge goes stale, cannot be cited, and cannot be permissioned.
4. Using long context as the standing replacement for a large knowledge base. Cost grows with size, and accuracy falls to context rot and lost-in-the-middle.
5. Adopting a graph because it is in fashion. If relationship questions are not the point, you pay 3 to 5 times as much and slow down simple lookups.
6. Treating a retrieval hit as a correct answer. Landing in the top list and being used correctly as evidence are separate checks.
7. Enforcing permissions after the answer is written. The only safe place is during retrieval.
8. Blaming the model for every failure. Most failures are already fixed, or already doomed, at parse, split, and store.
9. Trusting a model-written wiki or summary as if it were the source. Derived knowledge does not inherit authority.
10. Storing a generated context note as part of the original text. Keep them apart, and cite the original, so a bad note can be corrected.

Close

A good system is not a clever model. It is a reliable supply chain for evidence. Contextual retrieval answers lost context by writing it back onto the chunk. Graphs answer by storing the relationship itself. A wiki answers by compiling ahead of the question. Agentic retrieval answers by turning lookup into a loop of judgment.

Over the long run, the larger value is not a new retrieval algorithm. It is the ability to turn material into knowledge you can check, and to keep that knowledge maintained.

Sources

The figures below are the publishing authors' own test conditions. Any effect you rely on should be remeasured on your evaluation set.

• Anthropic, Introducing Contextual Retrieval (2024-09-19). Failure rates 5.7% → 2.9% → 1.9%, cost estimate, and the 200,000-token threshold.
• Anthropic, Contextual Embeddings Cookbook.
• Microsoft Research, From Local to Global: A Graph RAG Approach (2024-04).
• Microsoft Research, LazyGraphRAG (2024-11).
• Guo et al., LightRAG (2024-10).
• Sarthi et al., RAPTOR (ICLR 2024).
• Jina AI, Late Chunking (2024-08).
• Faysse et al., ColPali (ICLR 2025).
• Anthropic, Effective context engineering for AI agents (2025-09-29).
• Karpathy, LLM Wiki idea file (2026-04).
• Tobi Lütke, qmd.
• OWASP, RAG Security Cheat Sheet, on prompt injection and unauthorized document exposure.

This is a design proposal grounded in research and technical documentation. It is not a measured result on any one organization's files. Compare methods on a small set of your own material before you build the full system.