Why Context Windows Are the New Database

The way we feed information to LLMs is fundamentally a data architecture problem.

RAG is not a feature. It is database design for AI. Retrieval-augmented generation is just: store data, query data, inject data into context.

The context window as query result

When you prompt an LLM with retrieved documents, you are executing a query. The context window is the result set.

This means all the classic database problems apply: relevance ranking, index design, storage tradeoffs, query latency.

Dense vs sparse retrieval

Dense retrieval (vector embeddings) is great for semantic similarity. Sparse retrieval (BM25, keyword) is great for exact terms.

The best RAG systems combine both. Hybrid search with re-ranking captures both kinds of relevance.

What belongs in context

Not everything retrieved belongs in the context window. A good RAG pipeline has a filtering layer that evaluates relevance before injection.

Inject only what the model needs to answer the question. Context pollution degrades output quality faster than most engineers expect.

Memory as engineering discipline

AI memory architecture deserves the same rigor we apply to database schema design. The abstractions are different. The discipline is the same.

Design your memory first. Your prompts will get better automatically.