Shifting to RAG + MCP Agents in Production
RAG + MCP agents combine knowledge retrieval with live tool actions, and they are increasingly common in production AI systems. But shipping one to production surfaces failure modes that never show up in a demo. This article walks through five concrete failures documented by Sudip P. after deploying a RAG + MCP agent to 80 daily active users, and the fixes that followed.
Architecture and Concepts
The system routes each incoming query using a small LLM: does this request need knowledge retrieval (RAG), or a live tool action (MCP)? The RAG path combines BM25 keyword search with vector embeddings, then reranks the top 20 candidates with a cross-encoder. The MCP path validates tool inputs, enforces a 5-second timeout per call, and retries with exponential backoff, always returning a structured {ok, error, detail} response rather than a bare value. Both paths feed a frontier model for final synthesis, then pass through an evaluation and logging layer before the response reaches the user.
Five Things That Broke in Production
1. Retrieval Accuracy
Vector similarity alone retrieved factually incorrect chunks. "Tier-1 SLA for streaming" and "Tier-2 SLA for batch" read as semantically similar to an embedding model, but they mean completely different things operationally. A query about the streaming SLA could surface the batch SLA chunk and the agent would answer confidently with the wrong number.
2. Tool Output Validation
MCP tools returning null were interpreted by the model as success. When a job-status API timed out, the agent told users everything was fine while the system had actually been down for 90 minutes: the actual SLA was 30 minutes, but the agent cited 4 hours because it had no structured way to distinguish "no error" from "no data."
3. Routing Logic
A keyword-based router, 12 lines of if/elif statements, failed catastrophically against real user phrasing. A query like "hey is the prod thing borked again lol", or a 600-word Slack thread pasted in as context, broke the keyword matcher entirely. Replacing it with a small LLM router recovered accuracy without the cost of routing every query through a frontier model.
💡 Key takeaway: a small LLM router reached about 90% routing accuracy, at roughly 20 times lower cost and 3.6 times lower latency than routing every query through a frontier model.
4. Cost and Latency Explosion
With sequential model calls and no caching, daily spend ranged $300 to $400 for 80 active users. Latency followed the same pattern: P50 landed around 6 seconds, and P95 reached 14 seconds under load.
5. Missing Evaluation Framework
An embedding model upgrade silently degraded retrieval quality. Without a regression evaluation in place, the drop went unnoticed for three days, until users started complaining that answers had gotten worse.
Fixes and Recommendations
- Retrieval: combine vector similarity with a reranker (cross-encoder) rather than relying on embeddings alone to disambiguate near-identical phrasing.
- Tool outputs: enforce a structured
{ok, error, detail}response contract on every MCP tool, so the model can never mistake a missing value for a successful call. - Routing: use a small LLM router instead of keyword matching for anything user-facing; keep the frontier model for synthesis only.
- Cost and latency: parallelize independent calls where possible and cache repeated sub-queries instead of calling the frontier model sequentially for every step.
- Evaluation: run a regression evaluation suite on every model or embedding change, before it reaches production traffic.
Security and Governance Considerations
Beyond the five failures above, running RAG + MCP agents in production also raises standard security and governance questions that are easy to underestimate.
🚫 Insufficient Access Control: failing to implement proper access controls can lead to unauthorized access to sensitive data and systems. Fix: implement role-based access control (RBAC) and grant every user and service the minimum permissions needed to perform its task.
🚫 Inadequate Data Encryption: failing to encrypt sensitive data can lead to data breaches and unauthorized access. Fix: implement end-to-end encryption for sensitive data, both in transit and at rest.
🚫 Lack of Monitoring and Auditing: failing to monitor and audit system activity can lead to undetected security incidents and compliance issues. Fix: implement monitoring and auditing capable of detecting and responding to incidents in a timely manner.
Conclusion
None of these five failures are exotic. Retrieval ambiguity, silent tool failures, brittle routing, uncontrolled cost and latency, and the absence of a regression evaluation suite are all things that surface only once real users start sending real queries. Fixing them does not require a different architecture, it requires treating retrieval, tool calls, and routing as production systems with contracts, from the first day.
This article is based on the original post by Sudip P., "5 Things Broke When I Shipped a RAG + MCP Agent to Production", published on Towards AI.
