How to make a system answer from your data
The library metaphor, the words you need, and five conclusions to read before choosing a method for answering from your own material.
On This Page
Better answers do not come from a smarter model. They come from finding the evidence that matches the question and putting it in front of the model. This guide is about organizing, finding, and checking your material — not about writing a clever prompt and hoping.
It is written for people designing a system that answers from documents, tables, mail, and meeting notes that belong to one person or one organization. New terms are explained in place.
Four jobs, told as a library
Chunking splits a book into pieces a search can use. Embedding gives each piece a coordinate for its meaning, like a shelf number. Retrieval is the librarian pulling the relevant books. Generation is writing the report after reading those books.
Together, that basic pattern is retrieval-augmented generation (RAG): search first, then write. If any step is weak, the final answer is weak.
Words worth knowing first
| Term | Plain meaning |
|---|---|
| Token | The small unit a model reads. Cost and length are counted in tokens. |
| Chunk | One searchable piece of a longer document. |
| Vector | Meaning turned into a list of numbers. Similar meaning, similar numbers. |
| Vector database | A store that finds text with similar meaning quickly. |
| Keyword search (BM25) | Classic search for words that actually appear. Strong on names and codes. |
| Hybrid search | Meaning search and keyword search used together. |
| Reranking | Reading the shortlist against the question and ordering it again. |
| Knowledge graph | People, companies, and events drawn as dots, with relationships as lines. |
| Multi-hop | A question that needs several facts connected in order. |
| Fine-tuning | Training on examples to teach tone, format, or judgment habits. |
| Long context | Handing the model the whole corpus at once. |
| Agent | A way of working that picks tools, looks at results, and decides the next step. |
| Hallucination | A plausible statement with no support in the material. |
| Citation | Showing which passage each claim came from. |
Do not start by picking a brand-name method
Look at the shape of the material and the questions people actually ask. The structure usually follows.
- A small corpus may not need a search system. Under about 200,000 tokens — roughly 500 pages — and rarely changing, putting the whole set in a long context is often simpler, cheaper, and more accurate.
- If most questions have an answer in one place, a careful basic stack is enough. Hybrid search, a short context note on each chunk, and reranking is the best-tested default. In Anthropic's published test, that stack cut the rate of missing the right chunk in the top 20 from 5.7% to 1.9%.
- Questions that connect several documents need more than chunk search. A graph, or a human-readable wiki compiled ahead of time, is the right family. Graphs can be worse than basic search on simple facts. Use them when the connection is the point.
- Tone, format, and judgment are not search problems. Fine-tuning teaches behavior. Using it to memorize facts that change leaves you with stale knowledge and no citation. Facts come from retrieval. Behavior comes from training.
- You cannot improve what you do not measure. Start with 50 to 100 real questions and the passages that answer them. Changing chunk size by feel is a gamble.
Three more points belong next to those conclusions.
- Longer input is not the same as better recall. Context rot — models missing material as the input grows — shows up in current models.
- Every method is really a choice about when you pay to organize: at ingest time, or at question time.
- Questions that count or sum, such as "how many contracts last quarter," belong to a database query, not to search.
How the guide is arranged
- Why basic retrieval fails
- Preparing the material
- Finding evidence and answering from it
- Graphs and a compiled wiki
- Long context, agents, and tools by data shape
- A hybrid design and where to start
- Measurement, operations, and what not to do
This is a design proposal grounded in published research and technical write-ups. It is not a benchmark on any one organization's files. Published numbers show direction. The size of the effect has to be remeasured on your own evaluation set.