Building Customer Support AI Agents at 100M-User Scale: An Evaluation-Driven Framework

Aman Gupta, Kevin Rossell, Edesio Alcobaça, Jose Chrystian Lima Pacheco, Carolina Baptista de Lima, Shao Tang, Luiz Paulo Rabachini, Luis Moneda, Herbert Fei, Daniel Silva, Rohan RamanatharXiv 2026 · DOI · arXiv

TL;DR

Nubank presents a unified, evaluation-driven framework for deploying customer support AI agents to over 100 million users. By combining modular context engineering with a calibrated, automated LLM-as-a-judge pipeline, they achieved massive gains in customer satisfaction and automation, bringing AI performance within striking distance of human experts.

What Problem Does This Paper Address?

While general-purpose AI agents excel at coding or open-ended research, deploying them for Customer Support (CS) in a highly regulated financial environment introduces a much higher quality bar. Mistakes can cause direct financial losses and permanently erode customer trust.

CS agents must handle sensitive data securely, operate within narrow but highly complex domain boundaries, and gracefully hand off conversations to humans when confidence is low. Traditionally, prompt engineering, evaluation, and online deployment are treated as isolated silos, leading to hidden bugs that only surface post-deployment. This paper addresses how to build a reliable, predictive offline evaluation pipeline that directly accelerates production iteration velocity.

Figure 1: Architecture of a Customer Support AI agent system. User queries arrive via multiple channels and are handled autonomously, with fallback to human agents under low confidence.

Methods

The authors developed a production-tested system architecture centered around two main pillars:

  • Agent Architecture: Built on the ReACT (Reason + Act) paradigm, the LLM dynamically calls tools and reasons over results, allowing it to handle complex edge cases autonomously, unlike rigid, hardcoded workflows.
  • Modular Context Engineering: Instead of using monolithic prompts, the agent's context window is split into independently versioned modules: Instructions, Routines (executable translations of human Standard Operating Procedures), Macros (pre-written templates), Tool Specifications, and a Scratchpad (working memory).
  • Calibrated LLM-as-a-Judge: To evaluate the agents offline, the authors built an automated judge pipeline optimized via GEPA (Goal-driven Evolutionary Prompt Optimization) within the DSPy framework. The judge prompts are iteratively refined using human analyst rationales to maximize alignment with human judgment.
  • Dual-Loop Lifecycle: A fast loop uses offline simulations and optimized judges to catch failures and refine prompts before deployment, while a slow loop feeds online business metrics back into the core architecture.
Figure 2: Overview of our evaluation-driven development framework. Four stages—context engineering, prompt iteration, offline evaluation, and production deployment—are connected by a feedback loop where online metrics inform subsequent iterations. The correlation between offline evaluation and production outcomes validates the pipeline’s predictive power.

Datasets

The authors constructed five domain-specific evaluation datasets (E1–E5) from real, anonymized Nubank customer support interactions:

  • E1: Agent Reissue Failure (888 samples)
  • E2: Customer Input Verification (90 samples)
  • E3: Card Delivery Data Check (816 samples)
  • E4: Response Conciseness (246 samples)
  • E5: Resolution Completeness (90 samples)

Ground-truth labels and detailed textual justifications were provided by three domain-expert operations analysts, using a majority vote to establish the final metrics.

Key Insights and Ideas

  • Evaluation Quality Dictates Iteration Velocity: A reliable offline evaluation pipeline is the ultimate bottleneck. If your offline evals accurately predict online success, teams can iterate rapidly without waiting for slow, expensive online A/B tests.
  • More Capable Models Need Stricter Prompts: When upgrading from GPT-4.1 to GPT-5, the authors found that more powerful models followed instructions so strictly that they stopped inferring implicit context (e.g., failing to call tools before responding unless explicitly told to do so). This requires a shift toward explicit "tool-first" principles.
  • Manual Prompts Can Degrade Performance: In highly imbalanced datasets, a naive human-written starter prompt actually performed worse than a simple majority-class baseline. Automated prompt optimization (GEPA) was required to align the LLM judge with expert human logic.
  • Prompt Optimization Drives Model Convergence: Optimizing the judge rubrics caused wildly different model families (GPT-4o, GPT-5, o3) to converge on the same scores, raising pairwise Cohen's Kappa from near-zero to $\kappa > 0.7$, making the evaluation architecture-agnostic.

Key Results

The framework was validated across 5 production use cases: Card Delivery, Debt Management, Credit Limit, Card Management, and Product Explainer.

  • Headline Gains: In the Card Delivery deployment, large-scale A/B testing showed a 37 percentage-point (p.p.) absolute increase in transactional Net Promoter Score (tNPS) and a 29 p.p. increase in the self-service rate (SSR) compared to legacy variants.
  • Human Parity: Across 4 out of 5 domains, the AI agent's customer satisfaction (tNPS) landed within 1 to 10 p.p. of expert human agents.
  • Predictive Power: The experiments proved a strong positive correlation between offline evaluation failure reduction and online tNPS/SSR gains, validating the predictive power of the framework.
Figure 3: Online A/B-test gains for five production agents built with the same eval-driven framework. AI transactional NPS (tNPS) improves across all five domains; self-service rate improves in four (Product Explainer’s slight dip stems from transient LLM request failures). On most use cases the resulting AI tNPS lands within a few percentage points of expert human agents (Table 3).

Limitations and Future Work

  • The Human Gap in Complex Domains: While simpler domains neared human parity, highly nuanced domains like Debt Management still show a massive gap (−23.6 p.p. tNPS compared to humans), requiring deeper emotional intelligence and complex numerical reasoning.
  • Infrastructure Fragility: The Product Explainer agent suffered a slight dip in self-service rate due to transient LLM request failures, highlighting the dependency on upstream API stability.
  • Future Directions: The team plans to expand use case coverage, test open-source model alternatives, and use this evaluation-driven data curation to close the loop on fine-tuning proprietary models.

Summary updated

Abstract

The rapid rise in LLM capabilities has made AI agents increasingly viable across a broad range of tasks. Among the most promising applications is building production-ready customer-facing agents, a challenge that demands coordinated excellence in evaluation methodology, context engineering, training, and online measurement. Yet these critical pillars are typically developed in isolation, creating blind spots that only surface after deployment. In this paper, we present a unified framework that bridges offline development with online impact for customer support AI agents at Nubank, a company with 100M+ users. Our approach integrates several key components: (1) structured context engineering tailored to customer support agents, (2) systematic human-in-the-loop prompt iteration, (3) rigorous LLM judge evaluation with measured inter-rater agreement and GEPA optimization for consistency, and (4) ideation-to-production validation. A central insight is that evaluation-pipeline quality directly determines iteration velocity. We present results from five production deployments spanning distinct domains: card delivery, debt management, credit-limit support, card management, and product explanation. These deployments deliver consistent customer-satisfaction gains while substantially accelerating iteration. In our card-delivery deployment, large-scale A/B testing yields a 37 percentage-point improvement in AI transactional Net Promoter Score and a 29 percentage-point gain in self-service rate over prior agent variants, alongside a strong correlation between offline simulation metrics and online outcomes, demonstrating that eval-driven development reliably predicts production impact. On most use cases, AI satisfaction reaches within a few percentage points of expert human agents.
BibTeX
@misc{gupta2026building,
  author = {Aman Gupta and Kevin Rossell and Edesio Alcobaça and Jose Chrystian Lima Pacheco and Carolina Baptista de Lima and Shao Tang and Luiz Paulo Rabachini and Luis Moneda and Herbert Fei and Daniel Silva and Rohan Ramanath},
  title = {{Building Customer Support AI Agents at 100M-User Scale: An Evaluation-Driven Framework}},
  year = {2026},
  eprint = {2606.08867},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.08867},
}

Loading related content...

Figure