Presentation: From AI Agent Demo to Production: Automated Testing and Evaluation

A summary of Zhou (Jo) Yu's QCon AI Boston 2026 talk on getting AI agents out of the demo stage with simulation-driven automated testing and evaluation. Zhou Yu is a Professor of Computer Science at Columbia University and the founder of Arklex.ai.

Why Agents Stay in the Demo

• Many agents never leave the demo stage: the speaker cites a figure of 95% of agents staying in demos.
• In regulated fields such as finance and healthcare, she says the bottleneck is compliance.
• Single-turn, static benchmarks do not fit multi-turn agents. Tool outputs change over time (an account balance, for example), so fixed reference answers fail.
• Manual testing by friends or coworkers is slow, has poor coverage and does not reflect real user intent.

She also gives an example of what compliance means in practice: under regulation, a credit card cannot be opened without the customer authorizing it themselves by clicking a link, which is why a voice credit card agent shown in the talk remains a demo. E-commerce carries less risk and has deployed agents more often.

Simulation-Driven Testing

The proposal is to let AI agents play users with different personas, which gives a wider range of realistic behavior than internal testers. Each scenario combines three parts: a user profile, user goals derived from the tools and capabilities the agent has enabled, and context information such as documents. The simulated user talks to the product agent, calls APIs and can take GUI actions such as clicking product cards.

Building a Scenario

A scenario is the unit of a test. Permuting profiles, goals and context gives broad coverage, and each scenario maps to one simulated conversation.

• User profile. Attributes set by the product or operations team, such as demographics or how budget conscious the user is.
• User goals. Intents seeded by the capabilities the agent has enabled, and their sub-goals.
• Context information. Product documents or goal-related information.
• Simulated users are exposed as an endpoint, so a test can be launched on demand with an API call, and manual testing is needed only at the end.

How the Evaluation Works

• Task completion: checks verify the end state in backend systems, for example whether a return was actually processed, rather than trusting the agent's claim.
• LLM-as-judge: scores turn-level qualities such as helpfulness and brand voice, grounded with hard negatives and positives.
• CI/CD: fixed scenario sets run on each build, and a threshold on the conversation score decides pass or fail.
• Evolving golden set: production logs reveal gaps in the scenario distribution, and those gaps become new scenarios.
• Cold and warm start: with no data, domain experts are interviewed to shape the first scenarios; with existing data, the simulations match its distribution.

A Worked Example

In the talk's shopping agent example, the agent asks which of two orders the user means: order 1299, already fulfilled, or order 1300, still pending. The simulated user then asks when 1300 will ship, asks for an arrival estimate and finally asks for a customer service email. The agent fails to provide shipping information, which points to a possibly broken shipping tool or a gap in the RAG or vector database. Because the scenario is known in advance, the check can be this fine-grained.

What the Report Contains

In the ArkSim demo, a report is produced per run. It scores five dimensions and classifies errors into five categories, with an explanation and a suggested fix for each error.

Each conversation also has its own view with turn-level error highlights, the user goal, the simulated user profile and a goal completion explanation.

Wiring It into CI/CD

1. A code push triggers a build that starts the new version of the agent.
2. The simulator runs the fixed set of conversations. Configuration directories, the number of conversations and the number of turns can be overridden.
3. The conversations are evaluated, and a threshold on the conversation score decides pass or fail.
4. The report gives an earlier readiness signal and reduces manual testing time.

Cost matters: the speaker says running 10,000 conversations per CI cycle could cost a couple of thousand dollars (the transcript is auto-generated, so check the figure in the video), which is why the simulator is also judged on using the fewest scenarios for the most coverage.

Measuring the Simulator and the Agent

The talk distinguishes metrics for the simulator from metrics for the agent. For the simulator: tool-call distribution entropy, tool-call transition entropy, trajectory distance, realism, cost (the fewest scenarios for the most coverage) and agent failure identification. For the agent: customer needs completion, efficiency and call-to-action completion. In the demo, the judge dimensions were helpfulness, coherence, verbosity, relevance and faithfulness.

Keeping the Golden Set Current

Production logs reveal distribution shifts, such as a holiday promotion or a new policy. The proposed approach compares production logs with the simulated conversations to find gaps and then generates new scenarios across user attributes, goals and context. The speaker calls maintaining a golden set a huge effort for operations teams, and automated updating is meant to reduce it.

Examples from the Talk

• Walmart Sparky and Amazon Rufus are cited as shopping agents in production.
• A credit card voice agent is shown as a demo that is not in production because of compliance rules, such as requiring the customer to authorize the application through a link.
• Pearson, an education company, is cited as a two-year customer; simulation helps test across more than 200 countries with different policies.
• In a mortgage underwriting workflow, synthetic documents (including corner cases and imperfect scans) are generated because real data is restricted by PII rules.
• In an e-commerce corner case, a return request fails because the item has not shipped, so the flow should fall back to cancelling the order.

What Simulation Can Fix

The return versus cancel example is a design lesson. A rigid flow routes "return order" to a return tool, and the conversation fails when there is nothing returnable because the item has not shipped. The better design falls back to cancellation. Simulated users surface corner cases like this, and the output can suggest prompt or flow changes. The same synthetic data can fine-tune smaller models by rewarding successful conversations and penalizing unsuccessful ones.

Workflow Agents and Self-Learning

• Mortgage underwriting. Real documents are restricted by PII rules, so synthetic users act as digital twins. The pipeline generates consistent attributes, filled forms, synthetic pay stubs and bank statements, and underwriting corner cases from handbooks, then adds realistic defects such as missing pages, skew, blur and poor lighting.
• Self-learning agents. Simulation serves as a surrogate for real users before deployment. After deployment, tools, memory structure and prompts are iterated with real production data.

Arklex's open-source simulator is called ArkSim.

Questions from the Audience

• Edge-case personas, such as a toddler or a confused elderly caller. Simulation can cover them using existing data and domain knowledge, with expert review to refine the personas.
• Soft goals, such as product recommendations. A model estimates whether the user would accept a recommendation from their history. Acceptance signals good performance and rejection signals room for improvement.
• How accurate are the persona behaviors? Cold start, with no data, is the hardest case. Experts are interviewed to build the first scenarios, which are refined through simulation, feedback, deployment and real data. A warm start uses production data to match the real distribution.

Recommendations

• Run simulations early, before manual testing, and use them as a surrogate for real users.
• Ground evaluation in specific task criteria and verify outcomes in backend systems.
• Have domain experts review simulated conversations and feed back hard negatives.
• Avoid rigid flows: use simulation to find corner cases and refine prompts and flows.
• Keep the scenario set current using production logs.
• Use simulated data to fine-tune smaller models for high-volume customer-facing agents, and to train workflow agents where real data is restricted.

Sources

• From AI Agent Demo to Production: Automated Testing and Evaluation (InfoQ presentation)