InteractionBench: Measuring Timing, Task Accuracy, Whisper and Volume, and Grounding in Real-Time Voice Systems

Richard Yucheng He, Chen Xu, Yihang Liu, Tairan Chen, Baodong CaoarXiv 2026 · arXiv

TL;DR

InteractionBench is a new, unified evaluation framework designed to put real-time, "full-duplex" voice AI through its paces. By testing seven major systems—including GPT-Realtime and Gemini Live—across timing, task accuracy, paralinguistics, and long-term grounding, the paper reveals a significant performance gap between hosted commercial models and open-source alternatives, while highlighting that even the best systems struggle with proactive interruptions like real-time grammar correction.

What Problem Does This Paper Address?

Traditional voice assistants operate in a "walkie-talkie" mode: you speak, you stop, and then the AI responds. Modern "full-duplex" systems, however, are designed to listen and speak simultaneously, handling interruptions, backchannels (like "mhm" or "okay"), and mid-sentence pauses.

Until now, benchmarking these systems has been fragmented. One test might measure how quickly a model stops talking when interrupted, while another tests its ability to remember a grocery list, often using different datasets and protocols. This makes it nearly impossible to compare systems holistically. InteractionBench addresses this by providing a single "harness" that evaluates four critical dimensions of interaction using the exact same audio stimuli for every system.

Methods

The researchers built an evaluation harness that streams pre-recorded user audio to various AI systems in real-time.

Figure 1: The evaluation harness. Every system hears the same recorded user audio through its own driver, and both tracks are recorded on one timeline, transcribed, and scored by the same graders.

Key methodological innovations include:

  • Unified Timeline: Both user and system audio are recorded on a single two-track timeline, allowing for millisecond-level precision in measuring turn-taking and interruptions.
  • Timeline Reconstruction: For systems like OpenAI’s GPT-Live, which omit silent frames in their stream, the authors developed a method to rebuild the timeline based on arrival timestamps to ensure fair timing measurements.
  • Hybrid Grading: The benchmark uses deterministic rules for timing and acoustic analysis (e.g., detecting whispers via harmonics-to-noise ratios) and an LLM judge (Gemini 2.5 Flash) to evaluate the semantic correctness of answers and conversational grounding.

Datasets

The benchmark consists of 221 items per system across 13 distinct scenarios.

  • Stimuli: User speech was synthesized using MiMo TTS, including variations in accent, volume (whispering/shouting), and style.
  • Scenarios: These include timing tests (pauses, interruptions), spoken tasks (logic puzzles, countdowns), and paralinguistic control.
  • Groundedness Probes: The authors generated 10 long-form conversations (22–24 turns each) containing 52 specific "probes" designed to test if the AI can maintain a consistent state, reject false accusations, and flag nonsense questions.
InteractionBench covers thirteen scenarios across four dimensions, from millisecond timing to long-term conversational memory.

Key Insights and Ideas

  • The Full-Duplex Correlation: Among native full-duplex models, there is a strong correlation ($\rho = 0.83$) between timing and task accuracy. Systems that are better at knowing when to speak are generally better at knowing what to say.
  • The "Network Delay" Illusion: Timing measured at the client (where the user is) includes network latency. For GPT-Live, the authors found that removing a ~1.5s network delay actually lowered its timing score from 86% to 56%. Without the delay, the model frequently started speaking before the user had finished their sentence.
  • The Proactivity Gap: No system successfully passed the "Grammar" scenario, which required the AI to interrupt the user immediately upon hearing a grammatical error. This suggests that current models are still largely reactive rather than truly proactive.
  • Whispering is a Rare Skill: Despite the capability being marketed, only Gemini Live whispered reliably (8/10 items). Most other systems responded to whispered requests in a normal voice.

Key Results

The evaluation compared hosted APIs (GPT-Live, GPT-Realtime, Gemini Live), open models (Moshi, PersonaPlex, Freeze-Omni), and a traditional turn-based "cascade" (ASR + LLM + TTS).

  • Task Accuracy: The hosted systems and the turn-based cascade were neck-and-neck, passing 68–76% of tasks. Open full-duplex models lagged significantly, with Moshi passing only 6%.
  • Groundedness: Hosted systems were far superior at maintaining conversational state, passing 67–88% of probes, compared to just 14–35% for open models.
  • Timing: GPT-Live achieved the highest timing score at the client level (86%), while Moshi showed the lowest median latency (190ms) but often spoke over the user inappropriately.
  • The Cascade Paradox: The turn-based cascade had the highest task accuracy (76%) but the worst timing for natural interaction, as it cannot handle pauses or interruptions within a turn.

Limitations and Future Work

The authors acknowledge several limitations:

  • Synthetic Speech: The use of TTS-generated audio lacks the "messiness" (background noise, stutters, emotional variance) of real human speech.
  • Instruction Sensitivity: Different systems required slightly different text prompts to function correctly, which may have influenced their behavior.
  • Narrow Scope: The benchmark does not yet measure "naturalness," safety, or long-horizon task completion (e.g., booking a flight).

Future work aims to incorporate real human speech, measure network delay across all hosted providers, and standardize instructions to further level the playing field.

Summary updated

Abstract

Most benchmarks for real-time voice systems focus on one aspect, such as turn-taking, instruction following, paralinguistic control, or conversational memory, each with its own data and set of systems. We present InteractionBench, a benchmark and evaluation harness that measures four dimensions on the same systems from the same recorded user audio: conversational timing, spoken-task accuracy, whisper and volume, and groundedness in conversations of more than twenty turns. The benchmark contains 221 items per system in thirteen scenarios. Before any system runs, we fix the expected floor-control actions for spoken tasks by rule and take the expected answers from scripted dialogues. We score whispering from the audio signal. We evaluate GPT-Live, GPT-Realtime, Gemini Live, Moshi, PersonaPlex, Freeze-Omni, and a turn-based cascade in 1,519 graded runs. Among the six full-duplex systems, timing and spoken-task accuracy are rank-correlated ($ρ= 0.83$), while the cascade has the highest spoken-task score (38/50) and the second-lowest timing score. No system corrects a user's grammar within the required window. Only Gemini Live whispers on most requests (8/10). Hosted systems pass 67-88% of groundedness probes, compared with 14-35% for open full-duplex models. Timing is measured where the client hears each system, so the timing of hosted systems includes network delay: for GPT-Live, whose server reports its own clock, removing a delay of about 1.5 s lowers its timing score from 86% to 56%. We also document where the protocol differs across systems, including the text instructions.
BibTeX
@misc{he2026interactionbench,
  author = {Richard Yucheng He and Chen Xu and Yihang Liu and Tairan Chen and Baodong Cao},
  title = {{InteractionBench: Measuring Timing, Task Accuracy, Whisper and Volume, and Grounding in Real-Time Voice Systems}},
  year = {2026},
  eprint = {2611.02430},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.02430},
}
Figure