AI in Production

AI in Production: Panellists Highlight Silent Failures, Harness Engineering, and Agent Governance

A summary of the InfoQ webinar on AI in production: why AI fails silently, how to review code that agents write, and what agent governance needs, with views from four practitioners.

AI in Production: Panellists Highlight Silent Failures, Harness Engineering, and Agent Governance

A summary of the InfoQ webinar "AI in Production: What Breaks, What Works, and Who Approves It?", reported by Olimpiu Pop on 16 October 2026. The panel covered silent failures, how to review code that agents write, and agent governance.

The Panel

Hien Luu moderated. The panellists were Zichuan Xiong, Prem Chandrasekaran (ThoughtWorks), Katharine Jarmul (privacy) and Lan Chu (financial AI). Luu opened with three drivers: AI fails differently from traditional software, generating convincing inaccuracies rather than crashing; coding agents produce production code faster than humans can review it; and sensitive data has new leak points in prompts, context windows and retrieved documents.

AI in production panel themesSilent failuressemantic drift (text-to-SQL)embedding drift (compliance agent)sensitive retrieval (RAG)memorizationGitSpawnHarness engineeringautomate style and syntax checkshuman review for architectureevaluate intent before PRsmutation testingevals as living infrastructureAgent governancered teaming, threat modelingstrict functional scopesdistinct non-human identitiestiered access: read vs reversible writeversioned MCP dependencies
Figure. The three themes of the panel and the examples each panellist gave, as summarized in the InfoQ report.

Silent Failures

  • Semantic drift (Xiong): text-to-SQL agents can flatten domain concepts, for example treating "member" the same way in sales and in customer support. This causes hidden analytics errors that pass standard tracing.
  • Embedding drift (Chu): a financial compliance agent blocked prohibited investment advice using vector similarity thresholds. An unannounced background update to the embedding model shifted the vector distance distribution, causing spikes in false positives and false negatives while infrastructure health checks stayed green.
  • Sensitive retrieval (Chandrasekaran): RAG pipelines can inadvertently retrieve sensitive documents.
  • Memorization (Jarmul): models memorize training data, which can enable de-anonymization or expose API keys.
  • GitSpawn (Chandrasekaran): a vulnerability where malicious repository configurations can execute unauthorized code during routine commands.
Embedding drift failure chainUnannounced backgroundupdate to the embeddingmodelVector distancedistribution shiftsSimilarity thresholds misfire:false positives and falsenegatives in the compliance agentInfrastructure health checks stayed green throughout.
Figure. The embedding drift case reported by Lan Chu: the failure chain, while infrastructure health checks stayed green.

Harness Engineering and Code Review

  • Line-by-line manual review cannot keep up with agent output. Chandrasekaran recommends automating style and syntax checks in CI/CD and reserving human review for architecture, business intent and non-functional requirements.
  • Xiong: teams should shift left and evaluate intent at the harness level before pull requests are generated.
  • Mutation testing (Chandrasekaran): deliberate code mutations show whether tests verify logic or only meet coverage metrics.
  • Evals (Chu): evals should be permanent, living infrastructure rather than pre-release checklists, with continuous online span-level tracing, log filtering and LLMs that categorize failure patterns. Production edge cases should feed back into offline benchmark suites.

Agent Governance and Security

  • Jarmul: do not rely on prompt engineering for defence. Use systematic red teaming, continuous threat modeling, strict functional scopes for autonomous agents, and observability that scales with event volume.
  • Chandrasekaran: treat every AI agent like a new engineering hire to limit its blast radius, with distinct non-human identities, tiered access controls separating read-only operations from reversible writes, versioned third-party Model Context Protocol (MCP) dependencies, and API secrets isolated outside the agent's runtime in sandboxes.

Engineering, security and privacy teams validate the safety guardrails, but the panel describes the approval to deploy as a strategic business decision.

Sources

Share

Subscribe for daily recipes. No spam, just food.