Joi-Bench-Task: Benchmarking Task Coordination in Full-duplex Voice Agents through Interactive Simulation

Xiangheng He, Mingjun Zhao, Zhehao Fan, Zhengjia Li, Xiaopeng Sun, Qingqing Xiao, Haolan Chen, Yu XuarXiv 2026 · arXiv

TL;DR

Joi-Bench-Task is a new benchmark designed to evaluate how well full-duplex voice agents coordinate complex tasks during evolving, real-time conversations. By using a controllable user simulator to trigger interruptions and request changes, the researchers found that even top-tier commercial agents struggle significantly, often omitting over 23% of required information after a simple interruption.

What Problem Does This Paper Address?

We are entering the era of "full-duplex" voice assistants—AI that can listen and speak simultaneously, allowing for natural "barge-ins" where a user interrupts to clarify or change a request. However, most existing benchmarks evaluate either conversational flow (can the AI handle a mid-sentence interruption?) or static task completion (can the AI book a flight?).

The "messy middle"—task coordination—remains largely untested. This involves tracking the state of multiple ongoing tools while the conversation shifts. For example, if an agent is halfway through reporting five flight options and the user interrupts to say, "Wait, only show me direct flights," the agent must:

  1. Stop the current report.
  2. Update the task requirements.
  3. Resume reporting without repeating what the user already heard or skipping the remaining valid options.

Current benchmarks don't sufficiently isolate these coordination failures, making it hard to see where agents lose the thread of the "execution vs. communication" state.

Methods

The authors developed a framework that pairs a voice agent with an interactive, controllable user simulator.

Figure 1: The framework for Joi-Bench-Task.

Key technical components include:

  • Tick-based Simulation Clock: To ensure reproducible results, the simulator decouples "conversational time" from "computation time." It processes audio in 200ms "ticks," pausing the simulation clock while the LLM generates a response so that processing lag doesn't affect the evaluation of agent latency.
  • Intervention Controller: This is the "brain" of the simulator. It monitors tool execution and agent speech (via incremental ASR) to trigger user behaviors at precise moments—such as interrupting exactly when the agent mentions a specific keyword.
  • Asynchronous Tool Environment: Tools have manually specified execution durations (e.g., 60 seconds for a "slow" search), creating a window for the user to ask for status updates or change requirements while work is still in progress.
Figure 2: User simulator architecture with a content-triggered inquiry–resume example.

Datasets

The benchmark consists of 60 scenarios across six everyday domains: in-car assistance, cooking, retail, travel, telecommunications, and household management. These scenarios are expanded into 140 specific test cases categorized into:

  • Coordination during execution: Testing status inquiries, priority changes, task withdrawals, and task additions while tools are running.
  • Coordination during result delivery: Testing "pause-resume" and "inquiry-resume" (interrupting with a follow-up question) while the agent is speaking.

Key Insights and Ideas

  • The Multi-Tasking Tax: All evaluated agents showed significantly higher latency when handling multiple tools compared to single-tool tasks. Handling a "bundle" of requests places a heavy cognitive load on the agent's ability to track what is running versus what is finished.
  • Hallucination under Pressure: A recurring failure pattern emerged where agents, after a request change, would "hallucinate" results for tools that hadn't even finished executing yet. This suggests a fundamental struggle in distinguishing between "completed work" and "work in progress."
  • The "What Have I Said?" Gap: Most agents failed to accurately track the "barge-in" point. If an agent sends a buffer of audio to the user, it often assumes the user heard all of it. If the user interrupts halfway through that buffer, the agent loses track of the exact word where the interruption occurred, leading to omitted information upon resumption.
  • Inquiry vs. Pause: Agents are much worse at resuming after a follow-up question than after a simple "pause" command. An inquiry introduces a new conversational goal that often causes the agent to "forget" its previous obligation to finish the original report.

Key Results

The researchers evaluated five leading commercial agents (including GPT Live 1.0, Gemini 3.8 Live, and Grok Voice 3.0). The results highlight a major gap between conversational fluency and task coordination:

  • Request Change Failures: Under request changes (like adding or canceling a task mid-stream), the highest result completion rate was only 55%.
  • Information Omission: After being interrupted, four out of five agents missed significant chunks of information, with average omission rates of at least 23%.
  • Resumption Rates: While GPT Live 1.0 excelled at resumption (90% rate), others like Realtime 2 struggled immensely, failing to resume unfinished reports in almost all cases (0% resumption after inquiry).
  • Sequential vs. Parallel: Some agents (like Realtime 1.5) defaulted to sequential tool calling, waiting for one task to finish before starting the next, which led to a "concurrency met" score of 0% and very high latencies.

Limitations and Future Work

  • Language & Culture: The current benchmark is limited to English. The authors plan to expand to multilingual evaluations to see if coordination patterns vary across different linguistic structures.
  • Acoustic Realism: The tests were conducted in "clean" audio environments. Real-world use involves background noise (traffic, kitchen sounds) which could further degrade an agent's ability to detect the exact timing of a barge-in.
  • Mock Tools: The environment uses mock functions with fixed durations. Future versions aim to integrate real-world APIs and databases to introduce naturally varying response times and data-dependent complexities.

Summary updated

Abstract

Full-duplex voice agents enable natural, hands-free interaction, allowing users to refine requests, redirect ongoing work, and seek clarification while the agent is speaking. However, existing voice-agent benchmarks insufficiently isolate coordination between task execution and conversation. We introduce a benchmark that assesses this coordination by examining whether agents track what is being executed, what has been completed, what has been communicated, and what remains to be reported as the conversation evolves. We propose an interactive user simulator that introduces configurable interventions at designated timing in tool execution and spoken result delivery, supporting controllable and reproducible evaluation conditions. We evaluate five leading commercial voice agents along three dimensions: execution responsiveness, coordination under request changes, and coordination under interruptions. Agent evaluations reveal persistent coordination failures. Under request changes, the highest result completion rate is only 55%, indicating difficulty for current voice agents adapting to new requests while preserving earlier obligations. After user interruption, four of the five agents exhibit average information omission rates of at least 23%. These findings highlight task coordination as a central challenge for current voice agents in complex, evolving conversations.
BibTeX
@misc{he2026joi,
  author = {Xiangheng He and Mingjun Zhao and Zhehao Fan and Zhengjia Li and Xiaopeng Sun and Qingqing Xiao and Haolan Chen and Yu Xu},
  title = {{Joi-Bench-Task: Benchmarking Task Coordination in Full-duplex Voice Agents through Interactive Simulation}},
  year = {2026},
  eprint = {2611.12118},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.12118},
}
Figure