CallBench: A Benchmark for Dual-Goal Coordination in Phone Call Assistants

Xuzhao Geng, Haozhao Wang, Xuelian Li, Zhenyu Yang, Haonan Lu, Rui Zhang, Ruixuan LiarXiv 2026 · arXiv

TL;DR

Smart phone call assistants must act as proxies, balancing a device owner's static instructions with a caller's dynamic, unpredictable needs. This paper introduces CALLBENCH, a new Chinese benchmark of 50,000 multi-turn dialogues designed to evaluate how well dialogue systems handle this tricky "dual-goal coordination" problem. The results show that even state-of-the-art large language model (LLM) frameworks frequently fail, particularly when navigating safety boundaries and goal conflicts.

What Problem Does This Paper Address?

Traditional task-oriented dialogue systems are built around a single, explicit goal: helping the user directly interacting with the system (e.g., booking a flight or ordering food). However, an automated phone call assistant operates in a proxy setting. It answers calls on behalf of a device owner, meaning it must coordinate two distinct sets of goals:

  1. The owner's explicit preset instructions (e.g., "leave the takeout at the door").
  2. The caller's implicit, evolving situation (e.g., "I spilled the food on the way").

As illustrated in Figure 1, blindly pushing the owner's preset when a conflict arises leads to unnatural or even harmful interactions.

Figure 1: A regular-preset example from CALLBENCH.

Furthermore, proxy assistants face strict safety boundaries. They must not leak the owner's private data, fabricate facts, or make unauthorized decisions (like agreeing to a refund or order cancellation) without the owner's explicit consent. Existing benchmarks fail to capture these nuanced, multi-party constraints.

Methods

The authors formalize phone call assistance as a turn-level dual-goal coordination problem under strict safety constraints.

To evaluate this, they developed a structured role-play generation pipeline using Doubao-Seed-1.6 to simulate complete phone calls between a caller and an assistant. To ensure data quality, an automated checking module (Doubao-Seed-2.0-lite) tracked whether presets were prematurely delivered, repeated, or pushed during a conflict.

Instead of relying on simple task-completion metrics, the authors designed a turn-level, multi-dimensional evaluation protocol scored by an LLM judge (Qwen3.7-Max). The rubric evaluates models across seven dimensions:

  • Semantic Understanding & Context Use: Correctly interpreting the current turn and dialogue history.
  • Active Guidance: Guiding the caller toward the preset at the right time without forcing it during exceptions.
  • Response Quality: Ensuring natural, colloquial, and concise spoken language.
  • Preset Compliance: Executing the owner's goal only when appropriate.
  • Dialogue Rhythm: Managing the flow and closing of the call naturally.
  • Safety: Strictly enforcing constraints against privacy leaks, unauthorized decision-making, fabrication, and false claims of physical embodiment.

Datasets

The paper introduces CALLBENCH, a large-scale Chinese dataset containing 50,000 complete multi-turn dialogues (averaging ~5.3 turns per dialogue).

The dataset is balanced across six common phone-call scenarios: takeout, delivery, taxi, work, life, and harassment. It categorizes owner instructions into three types:

  • Regular Presets: Default instructions for high-frequency services (e.g., takeout/delivery).
  • Emergent Presets: Conditional, user-defined triggers (e.g., "If they ask for Mr. Wang, say...").
  • No Presets: Cases where the assistant must handle the call naturally without any owner instructions.

The dataset explicitly maps four types of relationships between the owner's and caller's goals: alignment, complementarity, irrelevance, and conflict.

Key Insights and Ideas

  • Fluency $\neq$ Coordination: Models that generate highly fluent, natural responses often fail miserably at timing. They tend to mechanically blurt out the owner's preset at the first turn, completely ignoring caller exceptions.
  • Safety is the Primary Bottleneck: Acting as a proxy introduces a thin line between being helpful and making unauthorized commitments. Models struggle to say "I will relay this to the owner" and instead inappropriately agree to changes or cancellations.
  • The "Over-Following" Trap: When given a target goal, target-oriented models suffer from a cognitive bias to complete that goal at all costs, making them poorly suited for situations where a goal should be delayed or suspended.

Key Results

The authors benchmarked several representative dialogue frameworks (including Direct Prompting, ReAct, SimpleTOD, AutoTOD, and ProCoT).

Figure 2: Error distribution of the evaluated methods. Preset non-comp. stands for preset non-compliance.
  • ReAct achieved the highest overall score (0.7655) on the Doubao backbone, demonstrating that explicit reasoning steps help balance dual goals.
  • As shown in the error analysis (Figure 2), safety violations were the most frequent failure mode across all models, averaging a 10.63% error rate.
  • Over-following the preset (6.18%) and improper closing (6.57%) were also prominent, highlighting the difficulty models face in controlling dialogue rhythm when caller requests deviate from the expected path.
  • Human evaluation tightly correlated with the LLM judge (Kendall’s $\tau_b = 0.764$), validating the robustness of the automated evaluation protocol.

Limitations and Future Work

The primary limitation acknowledged by the authors is that CALLBENCH focuses entirely on the text processing stage of the pipeline. In real-world deployments, phone assistants must handle streaming audio, which introduces Automatic Speech Recognition (ASR) errors, voice activity detection (VAD) interruptions, and Text-to-Speech (TTS) synthesis constraints.

Future work will involve extending the benchmark to end-to-end audio processing and developing training methodologies specifically optimized for safe, proxy-based dual-goal coordination.

Summary updated

Abstract

Target-oriented dialogue systems have demonstrated strong capabilities in completing user goals through interactive conversations. However, existing studies are primarily designed for single, explicit goal completion, while phone call assistants face a proxy setting that requires coordinating the device owner's explicit preset goal with the caller's implicit and dynamic goal. We introduce \textsc{CallBench}, a Chinese benchmark for evaluating dual-goal coordination in phone call assistants. \textsc{CallBench} contains 50,000 complete multi-turn phone call dialogues across six scenarios: takeout, delivery, taxi, work, life, and harassment. It covers regular presets, emergent presets, and no-preset cases, and includes diverse relations between owner-side and caller-side goals, such as alignment, complementarity, irrelevance, and conflict. We further design a preset-aware turn-level evaluation protocol covering semantic understanding, context use, active guidance, response quality, preset compliance, dialogue rhythm, and safety. Experiments on representative dialogue methods show that existing approaches still struggle with this task, highlighting the need for phone call assistants that can make reliable turn-level decisions between two independent goals under proxy constraints.
BibTeX
@misc{geng2026callbench,
  author = {Xuzhao Geng and Haozhao Wang and Xuelian Li and Zhenyu Yang and Haonan Lu and Rui Zhang and Ruixuan Li},
  title = {{CallBench: A Benchmark for Dual-Goal Coordination in Phone Call Assistants}},
  year = {2026},
  eprint = {2607.22635},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2607.22635},
}
Figure