TL;DR
Nubank presents a unified, evaluation-driven framework for deploying customer support AI agents to over 100 million users. By combining modular context engineering with a calibrated, automated LLM-as-a-judge pipeline, they achieved massive gains in customer satisfaction and automation, bringing AI performance within striking distance of human experts.
What Problem Does This Paper Address?
While general-purpose AI agents excel at coding or open-ended research, deploying them for Customer Support (CS) in a highly regulated financial environment introduces a much higher quality bar. Mistakes can cause direct financial losses and permanently erode customer trust.
CS agents must handle sensitive data securely, operate within narrow but highly complex domain boundaries, and gracefully hand off conversations to humans when confidence is low. Traditionally, prompt engineering, evaluation, and online deployment are treated as isolated silos, leading to hidden bugs that only surface post-deployment. This paper addresses how to build a reliable, predictive offline evaluation pipeline that directly accelerates production iteration velocity.
Methods
The authors developed a production-tested system architecture centered around two main pillars:
- Agent Architecture: Built on the ReACT (Reason + Act) paradigm, the LLM dynamically calls tools and reasons over results, allowing it to handle complex edge cases autonomously, unlike rigid, hardcoded workflows.
- Modular Context Engineering: Instead of using monolithic prompts, the agent's context window is split into independently versioned modules: Instructions, Routines (executable translations of human Standard Operating Procedures), Macros (pre-written templates), Tool Specifications, and a Scratchpad (working memory).
- Calibrated LLM-as-a-Judge: To evaluate the agents offline, the authors built an automated judge pipeline optimized via GEPA (Goal-driven Evolutionary Prompt Optimization) within the DSPy framework. The judge prompts are iteratively refined using human analyst rationales to maximize alignment with human judgment.
- Dual-Loop Lifecycle: A fast loop uses offline simulations and optimized judges to catch failures and refine prompts before deployment, while a slow loop feeds online business metrics back into the core architecture.
Datasets
The authors constructed five domain-specific evaluation datasets (E1–E5) from real, anonymized Nubank customer support interactions:
- E1: Agent Reissue Failure (888 samples)
- E2: Customer Input Verification (90 samples)
- E3: Card Delivery Data Check (816 samples)
- E4: Response Conciseness (246 samples)
- E5: Resolution Completeness (90 samples)
Ground-truth labels and detailed textual justifications were provided by three domain-expert operations analysts, using a majority vote to establish the final metrics.
Key Insights and Ideas
- Evaluation Quality Dictates Iteration Velocity: A reliable offline evaluation pipeline is the ultimate bottleneck. If your offline evals accurately predict online success, teams can iterate rapidly without waiting for slow, expensive online A/B tests.
- More Capable Models Need Stricter Prompts: When upgrading from GPT-4.1 to GPT-5, the authors found that more powerful models followed instructions so strictly that they stopped inferring implicit context (e.g., failing to call tools before responding unless explicitly told to do so). This requires a shift toward explicit "tool-first" principles.
- Manual Prompts Can Degrade Performance: In highly imbalanced datasets, a naive human-written starter prompt actually performed worse than a simple majority-class baseline. Automated prompt optimization (GEPA) was required to align the LLM judge with expert human logic.
- Prompt Optimization Drives Model Convergence: Optimizing the judge rubrics caused wildly different model families (GPT-4o, GPT-5, o3) to converge on the same scores, raising pairwise Cohen's Kappa from near-zero to $\kappa > 0.7$, making the evaluation architecture-agnostic.
Key Results
The framework was validated across 5 production use cases: Card Delivery, Debt Management, Credit Limit, Card Management, and Product Explainer.
- Headline Gains: In the Card Delivery deployment, large-scale A/B testing showed a 37 percentage-point (p.p.) absolute increase in transactional Net Promoter Score (tNPS) and a 29 p.p. increase in the self-service rate (SSR) compared to legacy variants.
- Human Parity: Across 4 out of 5 domains, the AI agent's customer satisfaction (tNPS) landed within 1 to 10 p.p. of expert human agents.
- Predictive Power: The experiments proved a strong positive correlation between offline evaluation failure reduction and online tNPS/SSR gains, validating the predictive power of the framework.
Limitations and Future Work
- The Human Gap in Complex Domains: While simpler domains neared human parity, highly nuanced domains like Debt Management still show a massive gap (−23.6 p.p. tNPS compared to humans), requiring deeper emotional intelligence and complex numerical reasoning.
- Infrastructure Fragility: The Product Explainer agent suffered a slight dip in self-service rate due to transient LLM request failures, highlighting the dependency on upstream API stability.
- Future Directions: The team plans to expand use case coverage, test open-source model alternatives, and use this evaluation-driven data curation to close the loop on fine-tuning proprietary models.
Summary updated