Screen Before You Serve: Simulation for Production Customer Experience AI Agents at 140M Scale
Nubank says simulated CX-agent trials tracked production outcomes closely enough to guide real deployments.
The paper describes a pre-deployment workflow using synthetic customers and simulated tool outputs to test multi-step support agents without touching live backends. On Nubank’s Card Delivery and Card Management agents, simulator scores correlated highly with production version-level binary evaluator scores across four deployed versions. Simulation-guided changes raised tNPS by 36.69 points in one live A/B test. A later screened open-weight setup, chosen after more than 16,000 simulated conversations, lifted self-service rate by 8.82 percentage points with no statistically significant tNPS change. ArXiv · AI/CL/LG's note
The paper describes a pre-deployment workflow using synthetic customers and simulated tool outputs to test multi-step support agents without touching live backends. On Nubank’s Card Delivery and Card Management agents, simulator scores correlated highly with production version-level binary evaluator scores across four deployed versions. Simulation-guided changes raised tNPS by 36.69 points in one live A/B test. A later screened open-weight setup, chosen after more than 16,000 simulated conversations, lifted self-service rate by 8.82 percentage points with no statistically significant tNPS change. ArXiv · AI/CL/LG's note
score 5