Put it all in, look it up live, or match the tool to the data

A third family drops the assumption that you must split and store everything first. Small corpora go in whole. Large ones are read on demand by an agent. Either way, you still choose what the model is allowed to see.

If it is small, put it all in

Anthropic's contextual-retrieval write-up says that under 200,000 tokens, about 500 pages, you should skip the retrieval system and put the corpus in the prompt. With prompt caching, repeat questions can cut latency by half or more and cost by up to 90%.

Research names the pattern cache-augmented generation (CAG): load the relevant documents into an extended context and reuse the cached computation. Search latency and search misses disappear. On bounded knowledge it has matched or beaten basic retrieval. On a commercial API, prompt caching plays the same role. The downsides are a rebuild whenever the material changes, and an expensive first request.

Longer is not clearer

Chroma's July 2025 context rot study found all 18 current models it tested getting worse as input grew. A toy "find the hidden sentence" task stays near perfect. Add similar-but-wrong distractors, or require understanding, and accuracy falls with length.

Lost in the middle — missing what sits in the center of a long prompt — is older. Anthropic describes this as an attention budget. Fitting is not the same as using. Long context is the right tool for reading one contract end to end. It is not a standing replacement for a large knowledge base.

Read it when you need it

Boris Cherny, who builds Claude Code, has said the early vector index was replaced because agentic lookup was generally better, simpler, and easier on security, privacy, and freshness. Core files stay in context. Everything else is found through file names, text search, and reading. Folder layout, filenames, and modification times are themselves signals.

The costs are clear. Exploring at runtime is slower than a precomputed index. A vague instruction sends the agent down dead ends and wastes context. The same exploration repeats on every question. One talk cited about 6,300 tokens to grep and read versus about 420 tokens for a stored chunk. That is an illustration, not a controlled benchmark.

The industry is not choosing "retrieval versus agents." It is letting an agent pick among keyword search, meaning search, file search, and database queries, then loop until it has enough.

• Self-RAG decides whether a search is needed and checks its own evidence while writing.
• Corrective RAG judges a weak result and rewrites the question or falls back to the web.
• Agentic RAG splits the question, picks a store per source, and stops when the evidence is enough.

More calls mean slower, costlier answers. Without a stop rule the loop never ends. Easy questions stay on a fixed path. Hard questions take the agent path. That split is the balance.

Connect to the source

The Model Context Protocol (MCP) is a shared way for a model to call outside systems. Drive, a CRM, an internal database, or Notion can be tools, so you query the original instead of copying it. The data stays current, and the source system's permissions still apply. You also inherit that system's search quality and rate limits. A practical mix keeps frequently asked material in a store or a wiki and reaches the rest live.

Different shapes need different tools

ColPali (2024, ICLR 2025) treats each page as an image and retrieves by that image. It beat text pipelines on visual document benchmarks. If you have few documents, handing page images to a multimodal model can be simpler.

Prices, inventory, dates, and totals lose accuracy the moment you chunk them. "Regional revenue last month" is SQL. Use a read-only account, a layer that defines what "revenue" means, and a check before the query runs.

One sentence separates the roles. Retrieval supplies knowledge, fine-tuning shapes behavior, and long context is working memory. Fine-tuning wins on format, voice, classification rules, and domain judgment habits. It loses when you ask it to memorize facts that change weekly: you cannot patch them, cite them, or apply per-person permissions. The usual stack is retrieval for facts, a system instruction for policy, and a light LoRA only when behavior still needs it.

When you build for real, keep at least these records separate.

Shared fields: tenant, access policy, created time, valid-from and valid-to, model version, and extraction-pipeline version. On a graph, the source and the valid time of a relationship matter most.