Design a hybrid system and choose where to start
Four layers of knowledge, a router that sends each question down a different path, a small-to-large stack, and starting points by scale.
On This Page
You do not need a full graph on day one. The point is not to buy several databases. It is to keep raw evidence, searchable text, relationships, and synthesized knowledge separate.
Four layers
| Layer | Holds | Analogy |
|---|---|---|
| 1. Raw evidence | Documents and data stored unchanged | The filing cabinet |
| 2. Searchable knowledge | Chunks, embeddings, keyword index, labels | The card catalog |
| 3. Structured knowledge | Entities, relationships, claims, events, time | A map of who is connected to whom |
| 4. Synthesized knowledge | Wiki, topic summaries, comparisons, analyses | The finished briefing |
Over time this stops being a search box and becomes a way to reuse experience and the reasons behind decisions. The second project can follow the wiki and the graph instead of rereading every old file, and update only what is new. The cost is coordination: when the wiki changes, the index, graph, summaries, and cache have to change with it.
A different path for each question
| Question | Send it to | Why |
|---|---|---|
| What are the termination terms of this contract? | Hybrid search plus parent-child | You need the clause, the exception, and the source together |
| What failure is common across projects? | Global graph search or the wiki summary layer | The question scans the whole set |
| What was last quarter's revenue? | SQL | It is a count or a sum |
| Review this report from start to finish | Long context | One document, read whole |
| How are company A and company B related, and what is in dispute? | Graph traversal plus source search | Find the link, then confirm in the text |
| Research this and write a briefing | Agentic exploration | Several steps |
One question often uses more than one path. The graph finds the relationship, hybrid search confirms the contract language, and SQL checks the amount.
In one sentence: keep document knowledge prepared in advance, query databases and business apps live, let the question type pick the tool, and answer with sources.
Start small, grow when the questions demand it
| Job | Start | Grow into |
|---|---|---|
| Original files | Object storage | Versioned storage |
| Labels and relationships | PostgreSQL | PostgreSQL plus an event log |
| Meaning search | pgvector | A dedicated vector database |
| Keyword search | PostgreSQL full text | Elasticsearch or OpenSearch |
| Graph | Relationship tables | A graph database such as Neo4j |
| Wiki | Markdown plus a database | A wiki with versions and approval |
| Parsing | A Python parser | A pipeline per document type |
| Jobs | A queue | Event processing such as Kafka |
| Evaluation | An evaluation set and a log | Automated regression and tracing |
PostgreSQL full text is a fine start and pairs with pgvector, but its ranking is not BM25. You also do not need Neo4j on day one. Relationship tables are enough while the graph is small. Move when the queries get deep. For Korean keywords, an analyzer with morphological analysis and a custom dictionary, such as Nori, is the better tool.
The methods side by side
| Method | Best questions | Scale | How often data changes | Difficulty | Cost per question | Main weakness |
|---|---|---|---|---|---|---|
| Long context plus cache | Almost any, on a small set | Under a few hundred thousand tokens | Rarely | Very easy | Many tokens, softened by cache | Accuracy fades with length; hard ceiling |
| Hybrid plus contextual retrieval plus rerank | Facts in one place | Unbounded | Often is fine | Medium | Low and fast | Global and multi-hop |
| Graph family | Global summaries, multi-hop, relationships | Medium to large | Depends on the variant | High | Varies widely | Simple facts; entity quality |
| LLM wiki | Synthesis and a growing personal body of knowledge | Hundreds of documents, one topic | Steady additions | Easy | Low to medium | Scale, permissions, accumulated error |
| Agent plus MCP | Live source systems, multi-step work | Unbounded | Real time | Medium | Slow, token-heavy | Latency; repeated exploration |
| Text-to-SQL | Totals, numbers, periods | Unbounded | Real time | Medium | Fast | Misread schema, bad queries |
| Page-image search | Slides, scans, charts | Many documents | Occasional | Medium | Medium, storage-heavy | Storage; weak exact text match |
| Fine-tuning | Format, voice, judgment habits | Not a corpus-size question | Needs retraining | Training cost | Fast at question time | Changing facts; no citations |
Where to start, by scale
One person or a small set. A few hundred documents, around 200,000 tokens: start with long context or a wiki. Building a vector store first is usually the wrong order. An agent maintaining a Markdown wiki can cite sources without a retrieval stack. Add a stronger search layer when you outgrow the index.
A team product. Thousands to tens of thousands of documents, questions every day: the improved search stack is the default. Structure-aware chunks, contextual retrieval, hybrid search, reranking, forced citations. Attach the evaluation set and a way to collect failed questions from the start. Add a graph only after you measure that relationship questions are actually common.
Large or regulated. Law, medicine, finance, and policy need the improved search stack plus structured-data tools, permission filters, and an audit trail. Open the agent path only for hard questions, and check citations on every answer.
Five questions set the order.
- Does the whole corpus fit in context and rarely change? Start with long context.
- Are most questions a fact inside one document? Use the improved search stack.
- Are relationships, multi-hop chains, or global summaries the actual value? Look at a graph or a linked wiki.
- Is this a person's growing notes, or an organization's official record? Wiki for the first, search plus governance for the second.
- Is the problem the model's behavior rather than the knowledge? Start with fine-tuning or instruction design.
The LaRA study at ICML 2025 points the same way. Whether retrieval or long context wins depends on the model, document length, retrieval setup, and question type. Neither side always wins.