⚡ The Problem RAG Alone Cannot Solve

Most RAG tutorials teach you the same thing: embed your documents, store them in a vector database, retrieve the top-k chunks at query time, and pass them to an LLM. The pipeline works beautifully in a notebook. Then you put it in front of real users and something breaks in a way no chunking parameter can fix.

The conversation runs for ten turns. The user references something they said three exchanges ago. The retrieved documents are relevant, but they are competing with a growing backlog of conversation history for space in the context window. The model starts truncating. It loses the thread. It hallucinates a fact that contradicts a document it saw four turns ago. Your system has a context management problem, not a retrieval problem.

Research bears this out with a striking statistic: nearly 65% of enterprise AI failures in 2025 were attributed to context drift or memory loss during multi-step reasoning, not to model capability gaps. A 2024 survey by Gao et al. found that over 70% of errors in modern LLM applications stem from incomplete, irrelevant, or poorly structured context - not from insufficient model capability. The bottleneck in 2026 has shifted from the model side to the context side.

🔬 Context Engineering: A New Discipline

In June 2025, Andrej Karpathy published a now-famous post on X that crystallised what many production engineers had been building implicitly for months. He wrote: "Context engineering is the delicate art and science of filling the context window with just the right information for the next step."

He was deliberate about calling it both an art and a science. Science because it involves task descriptions, few-shot examples, RAG, state and history, compression, tool definitions - all of which can be measured and optimised. Art because it requires intuition about how LLMs allocate attention, how much context aids versus confuses, and what order information should appear in.

It is worth drawing sharp distinctions between the three overlapping concepts:

"Context engineering is the delicate art and science of filling the context window with just the right information for the next step. Too little or of the wrong form and the LLM doesn't have the right context for optimal performance. Too much or too irrelevant and costs go up and performance comes down." - Andrej Karpathy, June 2025

As of March 2026, context engineering is no longer a standalone concept - it sits inside a broader agent stack that also includes agent harnesses, interoperability protocols (MCP), project memory for coding agents, and trace-first observability. The centre of gravity has shifted from "how to pack the best prompt" to "how agent systems manage runtime state, memory, tools, protocols, approvals, and long-horizon execution."

📅 The Three Generations of RAG Architecture

RAG is not a static pattern. It has evolved through three distinct architectural generations, each addressing the failures of the previous. Understanding this evolution explains why a naive RAG pipeline breaks in production - and what to build instead.

🔗 Understanding RAG Architecture in Depth

Before building the context engine layer, you need a firm grip on what RAG actually does and where each component makes decisions that affect context quality. A production RAG pipeline is not a single step - it is a cascade of decisions, each of which compounds into the final context the model sees.

Stage 1: Chunking Strategy

How you split your documents determines the fundamental granularity of information available for retrieval. The three canonical approaches have distinct trade-offs:

Stage 2: Hybrid Retrieval and Reciprocal Rank Fusion

Pure semantic (vector) retrieval casts a wide net but produces imperfect rankings. Relevant documents frequently end up "lost in the middle" of the result list. Production systems in 2026 combine two complementary signals: semantic embeddings (high recall, concept-level) and BM25/TF-IDF keyword matching (high precision, exact terminology). The scores are fused via Reciprocal Rank Fusion (RRF), which combines rankings without requiring score normalisation.

from langchain_community.vectorstores import Chroma
from langchain_community.retrievers import BM25Retriever
from langchain.retrievers import EnsembleRetriever
from langchain_openai import OpenAIEmbeddings

# Build both retrieval backends from the same documents
docs = load_documents()  # your document loading logic

# Semantic retriever - captures concept-level relevance
vectorstore = Chroma.from_documents(docs, embedding=OpenAIEmbeddings())
semantic_retriever = vectorstore.as_retriever(search_kwargs={"k": 25})

# Keyword retriever - captures exact terminology and proper nouns
bm25_retriever = BM25Retriever.from_documents(docs)
bm25_retriever.k = 25

# EnsembleRetriever applies Reciprocal Rank Fusion automatically.
# weights=[0.6, 0.4] means semantic scores weighted slightly higher.
# Tune toward 0.4/0.6 for keyword-heavy technical queries.
hybrid_retriever = EnsembleRetriever(
    retrievers=[semantic_retriever, bm25_retriever],
    weights=[0.6, 0.4]
)

# Returns fused, deduplicated results - typically top 15-20 candidates
candidates = hybrid_retriever.invoke("What is the difference between RAG and fine-tuning?")

⚙️ Context Engine: The Missing Layer

The context engine sits between retrieval output and the final prompt construction. Its job is to make explicit what naive RAG leaves implicit: exactly what information enters the context window, at what size, in what order, within a strict token budget. Without this layer, your system's behaviour under pressure (long conversations, noisy retrieval, latency constraints) is undefined.

Token Budget Allocation

Every context window has a fixed size. A context engine must make explicit allocation decisions before construction, not discover overflow at inference time. A practical allocation for a 128K token window looks like this:

import tiktoken
from dataclasses import dataclass, field
from typing import Optional

enc = tiktoken.get_encoding("cl100k_base")

def count_tokens(text: str) -> int:
    return len(enc.encode(text))

@dataclass
class ContextBudget:
    total_tokens: int = 128_000
    system_prompt_pct: float = 0.05
    recent_history_pct: float = 0.20
    summary_pct: float = 0.10
    documents_pct: float = 0.45
    output_reserve_pct: float = 0.20

    def docs_budget(self) -> int:
        return int(self.total_tokens * self.documents_pct)

    def history_budget(self) -> int:
        return int(self.total_tokens * (self.recent_history_pct + self.summary_pct))

class ContextEngine:
    def __init__(self, budget: ContextBudget):
        self.budget = budget

    def build_context(
        self,
        system_prompt: str,
        history_turns: list[dict],
        retrieved_docs: list[str],
        user_query: str,
        summary: Optional[str] = None
    ) -> dict:
        # 1. Allocate doc budget and compress if needed
        doc_budget = self.budget.docs_budget()
        docs_ctx, compression_ratio = self._fit_documents(retrieved_docs, doc_budget)

        # 2. Allocate history budget: recent verbatim + rolling summary
        hist_budget = self.budget.history_budget()
        hist_ctx = self._fit_history(history_turns, summary, hist_budget)

        return {
            "system": system_prompt,
            "context": docs_ctx,
            "history": hist_ctx,
            "query": user_query,
            "compression_ratio": compression_ratio,
            "tokens_used": count_tokens(docs_ctx + hist_ctx),
        }

    def _fit_documents(self, docs: list[str], budget: int) -> tuple[str, float]:
        # Pack documents greedily until budget is exhausted
        selected, total = [], 0
        for doc in docs:
            tokens = count_tokens(doc)
            if total + tokens <= budget:
                selected.append(doc)
                total += tokens
            else:
                break  # never overflow - hard budget ceiling
        original_tokens = sum(count_tokens(d) for d in docs)
        ratio = 1 - (total / original_tokens) if original_tokens > 0 else 0
        return "\n\n---\n\n".join(selected), ratio

    def _fit_history(self, turns: list[dict], summary: Optional[str], budget: int) -> str:
        # Always include summary first (if present), then most-recent turns
        parts, used = [], 0
        if summary:
            s_tokens = count_tokens(summary)
            if s_tokens <= budget // 3:  # cap summary at 1/3 of history budget
                parts.append(f"[Summary of prior conversation]\n{summary}")
                used += s_tokens
        # Add recent turns newest-first, stop when budget exhausted
        for turn in reversed(turns):
            text = f"{turn['role']}: {turn['content']}"
            t = count_tokens(text)
            if used + t > budget:
                break
            parts.insert(1 if summary else 0, text)
            used += t
        return "\n".join(parts)

💾 Memory Systems: In-Session vs Cross-Session

Memory in LLM systems is not a single concept - it is three distinct problems requiring three distinct architectures. Conflating them is one of the most common design mistakes in production systems.

📉 Compression Strategies and Token Budgets

When conversation history and retrieved documents together exceed the available token budget, the system must compress rather than truncate blindly. The difference matters: truncation discards arbitrarily. Compression preserves meaning at lower token cost. There are four production-grade compression strategies:

🏆 Re-ranking: Fixing Lost-in-the-Middle

Vector search maximises recall: it finds everything that might be relevant. The problem is that relevance at position #7 is just as invisible to the model as irrelevance at position #3. LLMs systematically attend less to information buried in the middle of long contexts - a well-documented phenomenon called "lost in the middle."

Re-ranking solves this by introducing a second-stage model that examines query-document pairs holistically. Cross-encoders (like BERT-based cross-attention models, Cohere Reranker, or ColBERT) evaluate relevance far more accurately than cosine similarity because they see the full text of both the query and the document simultaneously - not just their compressed embeddings.

🔎 Advanced Retrieval Patterns for Production

HyDE: Hypothetical Document Embeddings

Standard retrieval embeds the user's query and finds documents with similar embeddings. The problem: user queries are short, underspecified, and semantically far from the dense prose of the documents they are looking for. HyDE inverts this: it asks the LLM to generate a hypothetical answer to the query, then embeds that hypothetical answer and uses that richer embedding for retrieval. The hypothetical document is in the same semantic space as real documents, dramatically improving recall on sparse or ambiguous queries.

from langchain_core.prompts import ChatPromptTemplate
from langchain_anthropic import ChatAnthropic
from langchain_core.output_parsers import StrOutputParser
from langchain_community.vectorstores import Chroma
from langchain_openai import OpenAIEmbeddings

llm = ChatAnthropic(model="claude-haiku-4-5")  # cheap model for hypothesis generation
embeddings = OpenAIEmbeddings()
vectorstore = Chroma("my_index", embeddings)

# Step 1: Generate a hypothetical document that would answer the query
hypothesis_prompt = ChatPromptTemplate.from_template("""Write a short factual paragraph 
that would be a perfect answer to the following question. 
Be specific and detailed - do not hedge or say 'I don't know'.
Question: {query}
Hypothetical answer:""")

generate_hypothesis = hypothesis_prompt | llm | StrOutputParser()

def hyde_retrieve(query: str, k: int = 10) -> list:
    # Generate hypothetical document - used only for embedding, never shown to user
    hypothesis = generate_hypothesis.invoke({"query": query})

    # Retrieve using the hypothesis embedding (semantically richer than query alone)
    results = vectorstore.similarity_search_by_vector(
        embeddings.embed_query(hypothesis),
        k=k
    )
    return results

# Usage: sparse or ambiguous query benefits most from HyDE
# "What causes context rot in transformer models?"
# → hypothesis: detailed technical paragraph about attention dilution
# → retrieves docs that mention attention, context window, dilution
# → much better than embedding the 8-word query directly
results = hyde_retrieve("What causes context rot in transformer models?")

GraphRAG: Entity-Relationship Retrieval

Standard vector retrieval excels at pinpoint facts ("What does X mean?") but struggles with global questions ("What themes emerge across this entire corpus?"). GraphRAG, first demonstrated by Microsoft in 2024, builds an entity-relationship graph over the corpus during indexing. At query time, it traverses the graph rather than searching by embedding similarity, enabling theme-level and relationship-level answers with full traceability.

🤖 Agentic RAG and the Karpathy Pattern

In April 2026, Andrej Karpathy published a GitHub Gist describing an architectural pattern that drew immediate attention as a potential successor to both naive RAG and vector database approaches for personal and team knowledge management. He called it an LLM Knowledge Base - a persistent, agent-maintained wiki.

The core thesis: instead of using an LLM to perform just-in-time retrieval from a static pool of raw documents, deploy an LLM agent to proactively and continuously compile those documents into a persistent, interconnected, structured knowledge base - a wiki. The heavy cognitive work of reading, extracting entities, identifying relationships, and synthesising conclusions happens once at ingestion. Subsequent queries operate on curated knowledge, not raw documents.

📊 Measurement, Evaluation, and LLM-as-Judge

A context engineering pipeline without measurement is guesswork. Production RAG systems require systematic evaluation across four dimensions: retrieval quality, generation accuracy, system latency, and token cost. In 2026, LLM-as-judge has become the dominant approach for automated quality evaluation, supplementing but not replacing human annotation for high-stakes deployments.

from langchain_anthropic import ChatAnthropic
from langchain_core.prompts import ChatPromptTemplate
from pydantic import BaseModel, Field
from typing import Literal

class RAGEvalResult(BaseModel):
    faithfulness: Literal["yes", "no", "partial"] = Field(
        description="Does the answer contain only facts from the provided context?"
    )
    relevance: int = Field(ge=1, le=5, description="1=irrelevant, 5=perfectly relevant")
    completeness: int = Field(ge=1, le=5, description="1=major gaps, 5=fully addresses question")
    explanation: str = Field(description="One sentence explanation of the scores")

eval_prompt = ChatPromptTemplate.from_template("""You are an expert evaluator for RAG systems.
Given a question, a retrieved context, and a generated answer, evaluate the answer.

Question: {question}

Retrieved Context:
{context}

Generated Answer:
{answer}

Evaluate faithfulness (does the answer stick to context?), relevance, and completeness.
Return valid JSON matching the schema.""")

# Use claude-sonnet for evaluation (strong reasoning), haiku for generation (cost)
judge_llm = ChatAnthropic(model="claude-sonnet-4-5").with_structured_output(RAGEvalResult)

evaluator = eval_prompt | judge_llm

def evaluate_answer(question: str, context: str, answer: str) -> RAGEvalResult:
    return evaluator.invoke({
        "question": question,
        "context": context,
        "answer": answer
    })

# Run evaluation over your test set
results = [evaluate_answer(q, c, a) for q, c, a in test_cases]
avg_faithfulness = sum(1 for r in results if r.faithfulness == "yes") / len(results)
avg_relevance    = sum(r.relevance for r in results) / len(results)
print(f"Faithfulness: {avg_faithfulness:.1%}  Relevance: {avg_relevance:.2f}/5")

🛡 Security, Governance, and Anti-Patterns

⚖️ Choosing the Right Architecture

The right context architecture depends on your workload topology, latency budget, and knowledge base characteristics. The table below maps the key decision axes to the architectural pattern that performs best.

The context engine doesn't decide what to retrieve. It decides what the model actually sees. That distinction - between retrieving information and managing the context that shapes reasoning - is the architectural insight that separates production systems from demos.

All code examples target LangChain ≥ 0.3, Python 3.11+, and Claude / claude-haiku-4-5 as the LLM provider. HyDE retrieval uses langchain_community; hybrid retrieval uses EnsembleRetriever from langchain; token counting uses tiktoken (OpenAI). Cross-encoder re-ranking via Cohere Reranker or a local BERT cross-encoder (cross-encoder/ms-marco-MiniLM-L-6-v2).