OutboundEval: A Dual-Dimensional Benchmark for Expert-Level Intelligent Outbound Evaluation of Xbench's Professional-Aligned Series

Pengyu Xu, Shijia Li, Ao Sun, Feng Zhang, Yahan Li, Bo Wu, Zhanyu Ma, Jiguo Li, Jun Xu, Jiuchong Gao, Jinghua Hao, Renqing He, Rui Wang, Yang Liu, Xiaobo Hu, Fan Yang, Jia Zheng, Guanghua YaoarXiv 2025 · DOI · arXiv

Introduction

While Large Language Models (LLMs) have demonstrated exceptional proficiency in general-purpose dialogue, their application in professional "outbound calling" remains a significant challenge. Outbound calling requires an AI agent to initiate contact with a specific business objective—such as debt collection, recruitment, or technical support—while navigating complex business logic and unpredictable human behavior. Existing benchmarks often fail to capture these nuances, focusing instead on single-turn instructions or general chat.

To address this, we introduce VoiceAgentEval (also referred to as OutboundEval), a comprehensive benchmark designed to evaluate LLMs in expert-level outbound scenarios. Part of the Xbench Professional-Aligned Series, this framework provides a rigorous testing ground for multi-turn professional interactions across diverse business domains.

The VoiceAgentEval Framework

VoiceAgentEval moves beyond simple accuracy by employing a sophisticated architecture that mirrors real-world complexities. It integrates the entire pipeline from user simulation to multi-dimensional scoring.

Overview of the VoiceAgentEval framework, including the user simulator, evaluation dimensions, and business scenario scope.

1. Automated User Simulator

Rather than relying on static scripts, the framework utilizes a prompt-based LLM simulator (driven by GPT-4.1) that incorporates "Humanization Enhancements." These enhancements introduce emotional variability, memory lapses, and specific personality archetypes—such as "Resistant," "Hesitant," or "Analytical"—to challenge the agent's adaptability and resilience.

2. Dual-Layer Evaluation

The benchmark evaluates performance across two primary dimensions:

  • Task Flow Compliance (TFC): This metric assesses adherence to business logic. 70% of the score is derived from reaching "Key Nodes" (critical process steps), while 30% evaluates the correct sequence of operations.
  • General Interaction Capability (GIC): This dimension measures "human-likeness" across eight categories, including naturalness, coherence, hallucination handling, and emotional richness.

Dataset and Scenarios

The benchmark is built upon an authentic scenario-based corpus covering six major business domains: Customer Service, Sales, HR, Finance, Research, and Proactive Care. This is further divided into 30 sub-scenarios, such as bank account security risk control and fitness membership sales. With five distinct user types per sub-scenario, the benchmark comprises 150 unique evaluation instances.

Key Insights and Results

Testing across 12 state-of-the-art LLMs revealed several critical insights into the current state of AI outbound capabilities.

The Compliance-Fluency Trade-off

A notable finding was the divergence between rule-following and conversational charm. For instance, Claude-4-Sonnet excelled in Task Flow Compliance (TFC) but showed slightly lower scores in conversational naturalness. Conversely, models like Gemini-2.5-Flash demonstrated that smaller parameter counts do not necessarily limit interaction quality, sometimes outperforming larger models in the GIC dimension.

Leaderboard Performance

The overall leaderboard was topped by doubao-1.5-32k with a score of 0.8881, followed closely by GPT-4.1 at 0.8818. These models showed the most balanced performance across both task completion and interaction quality.

Pairwise comparison results showing AI win rates against human benchmarks in dialogue quality.

In "Turing-style" evaluations, AI-generated dialogues achieved an average human-likeness score of 8.27/9, nearing the human baseline of 8.50/9. Pairwise comparisons indicate that AI agents are increasingly capable of winning over human evaluators in specific dialogue contexts, with the AI winning 57.33% of the 150 tested instances.

Conclusion and Future Work

VoiceAgentEval establishes a new standard for evaluating professional-grade voice agents. While top-tier models are becoming nearly indistinguishable from humans in text-based simulations, future work must address the physical factors of real-time voice interaction, including network latency and ASR errors. Expanding the benchmark to include out-of-distribution business domains will further enhance its robustness as a tool for developing expert-level AI agents.

Summary updated

Abstract

We propose OutboundEval, a comprehensive benchmark for evaluating large language models (LLMs) in expert-level intelligent outbound calling scenarios. Unlike existing methods that suffer from three key limitations - insufficient dataset diversity and category coverage, unrealistic user simulation, and inaccurate evaluation metrics - OutboundEval addresses these issues through a structured framework. First, we design a benchmark spanning six major business domains and 30 representative sub-scenarios, each with scenario-specific process decomposition, weighted scoring, and domain-adaptive metrics. Second, we develop a large-model-driven User Simulator that generates diverse, persona-rich virtual users with realistic behaviors, emotional variability, and communication styles, providing a controlled yet authentic testing environment. Third, we introduce a dynamic evaluation method that adapts to task variations, integrating automated and human-in-the-loop assessment to measure task execution accuracy, professional knowledge application, adaptability, and user experience quality. Experiments on 12 state-of-the-art LLMs reveal distinct trade-offs between expert-level task completion and interaction fluency, offering practical insights for building reliable, human-like outbound AI systems. OutboundEval establishes a practical, extensible, and domain-oriented standard for benchmarking LLMs in professional applications.
BibTeX
@misc{xu2025outboundeval,
  author = {Pengyu Xu and Shijia Li and Ao Sun and Feng Zhang and Yahan Li and Bo Wu and Zhanyu Ma and Jiguo Li and Jun Xu and Jiuchong Gao and Jinghua Hao and Renqing He and Rui Wang and Yang Liu and Xiaobo Hu and Fan Yang and Jia Zheng and Guanghua Yao},
  title = {{OutboundEval: A Dual-Dimensional Benchmark for Expert-Level Intelligent Outbound Evaluation of Xbench's Professional-Aligned Series}},
  year = {2025},
  eprint = {2510.21244},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.21244},
}
Figure