TL;DR
Joi-Bench-Task is a new benchmark designed to evaluate how well full-duplex voice agents coordinate complex tasks during evolving, real-time conversations. By using a controllable user simulator to trigger interruptions and request changes, the researchers found that even top-tier commercial agents struggle significantly, often omitting over 23% of required information after a simple interruption.
What Problem Does This Paper Address?
We are entering the era of "full-duplex" voice assistants—AI that can listen and speak simultaneously, allowing for natural "barge-ins" where a user interrupts to clarify or change a request. However, most existing benchmarks evaluate either conversational flow (can the AI handle a mid-sentence interruption?) or static task completion (can the AI book a flight?).
The "messy middle"—task coordination—remains largely untested. This involves tracking the state of multiple ongoing tools while the conversation shifts. For example, if an agent is halfway through reporting five flight options and the user interrupts to say, "Wait, only show me direct flights," the agent must:
- Stop the current report.
- Update the task requirements.
- Resume reporting without repeating what the user already heard or skipping the remaining valid options.
Current benchmarks don't sufficiently isolate these coordination failures, making it hard to see where agents lose the thread of the "execution vs. communication" state.
Methods
The authors developed a framework that pairs a voice agent with an interactive, controllable user simulator.
Key technical components include:
- Tick-based Simulation Clock: To ensure reproducible results, the simulator decouples "conversational time" from "computation time." It processes audio in 200ms "ticks," pausing the simulation clock while the LLM generates a response so that processing lag doesn't affect the evaluation of agent latency.
- Intervention Controller: This is the "brain" of the simulator. It monitors tool execution and agent speech (via incremental ASR) to trigger user behaviors at precise moments—such as interrupting exactly when the agent mentions a specific keyword.
- Asynchronous Tool Environment: Tools have manually specified execution durations (e.g., 60 seconds for a "slow" search), creating a window for the user to ask for status updates or change requirements while work is still in progress.
Datasets
The benchmark consists of 60 scenarios across six everyday domains: in-car assistance, cooking, retail, travel, telecommunications, and household management. These scenarios are expanded into 140 specific test cases categorized into:
- Coordination during execution: Testing status inquiries, priority changes, task withdrawals, and task additions while tools are running.
- Coordination during result delivery: Testing "pause-resume" and "inquiry-resume" (interrupting with a follow-up question) while the agent is speaking.
Key Insights and Ideas
- The Multi-Tasking Tax: All evaluated agents showed significantly higher latency when handling multiple tools compared to single-tool tasks. Handling a "bundle" of requests places a heavy cognitive load on the agent's ability to track what is running versus what is finished.
- Hallucination under Pressure: A recurring failure pattern emerged where agents, after a request change, would "hallucinate" results for tools that hadn't even finished executing yet. This suggests a fundamental struggle in distinguishing between "completed work" and "work in progress."
- The "What Have I Said?" Gap: Most agents failed to accurately track the "barge-in" point. If an agent sends a buffer of audio to the user, it often assumes the user heard all of it. If the user interrupts halfway through that buffer, the agent loses track of the exact word where the interruption occurred, leading to omitted information upon resumption.
- Inquiry vs. Pause: Agents are much worse at resuming after a follow-up question than after a simple "pause" command. An inquiry introduces a new conversational goal that often causes the agent to "forget" its previous obligation to finish the original report.
Key Results
The researchers evaluated five leading commercial agents (including GPT Live 1.0, Gemini 3.8 Live, and Grok Voice 3.0). The results highlight a major gap between conversational fluency and task coordination:
- Request Change Failures: Under request changes (like adding or canceling a task mid-stream), the highest result completion rate was only 55%.
- Information Omission: After being interrupted, four out of five agents missed significant chunks of information, with average omission rates of at least 23%.
- Resumption Rates: While GPT Live 1.0 excelled at resumption (90% rate), others like Realtime 2 struggled immensely, failing to resume unfinished reports in almost all cases (0% resumption after inquiry).
- Sequential vs. Parallel: Some agents (like Realtime 1.5) defaulted to sequential tool calling, waiting for one task to finish before starting the next, which led to a "concurrency met" score of 0% and very high latencies.
Limitations and Future Work
- Language & Culture: The current benchmark is limited to English. The authors plan to expand to multilingual evaluations to see if coordination patterns vary across different linguistic structures.
- Acoustic Realism: The tests were conducted in "clean" audio environments. Real-world use involves background noise (traffic, kitchen sounds) which could further degrade an agent's ability to detect the exact timing of a barge-in.
- Mock Tools: The environment uses mock functions with fixed durations. Future versions aim to integrate real-world APIs and databases to introduce naturally varying response times and data-dependent complexities.
Summary updated