about
Paper: Nubank cuts AI agent update cycles from 3 weeks to under 5 days (arxiv.org)
2 points by rkovashikawa 12 days ago | hide | past | pdf | discuss on HN

In plain words: Fake customers chat with a support agent and its pretend tool responses, so teams can test thousands of conversations before real customers see anything. These trial scores tracked real-world scores closely, and the chosen setup lifted self-service by 8.82 points in a live test.

Abstract · Screen Before You Serve: Simulation for Production Customer Experience AI Agents at 140M Scale

Customer experience (CX) agents use tools and large language models to address customer requests and guide conversational interactions with an organization's products. Improving these agents, especially in regulated industries, is difficult: they must detect intent, follow complex operational policies and use tools reliably. Manual end-to-end testing offers limited coverage, while live experiments expose customers to failures that can erode trust. We present a hypothesis-driven simulation workflow for screening candidate CX agents before deployment. Synthetic customers react to agent responses and simulated tool outputs enable multi-step agentic workflows without invoking production backends. We use the Snowglobe simulator on Nubank's Card Delivery agent and its expanded successor, Card Management - Nubank's highest-volume chat-support agent in Brazil. Across 4 deployed versions, simulated and production version-level binary evaluator scores show high correlation. Simulation-guided iteration increased transactional net promoter score (tNPS) by 36.69 points in a live A/B test. We also screened open-weight configurations in over 16,000 simulated conversations. In a subsequent live A/B test, the selected model increased self-service rate (SSR) by 8.82 percentage points to the highest level observed at Nubank, with no statistically significant change in tNPS. Simulation made broad exploration of models, reasoning settings, and prompts feasible without customer exposure, enabling production improvements that would have been impractical to pursue through live experimentation alone.

Edesio Alcoba, Kevin Rossell, Aman Gupta, Shao Tang, Jiwoo Hong, Pabel Carrillo-Mendoza, Wanderson Conceição Ferreira, Alvaro Tedeschi, Zayd Simjee, Shreya Rajpal, Bruno Finardi Hime, Christian Sousa, et al.
arXiv:2609.30137 · cs.AI, cs.CL · submitted Sep 24, 2026
abstract · pdf · html · 17 pages, 11 figures

add comment on HN