TL;DR
InteractionBench is a new, unified evaluation framework designed to put real-time, "full-duplex" voice AI through its paces. By testing seven major systems—including GPT-Realtime and Gemini Live—across timing, task accuracy, paralinguistics, and long-term grounding, the paper reveals a significant performance gap between hosted commercial models and open-source alternatives, while highlighting that even the best systems struggle with proactive interruptions like real-time grammar correction.
What Problem Does This Paper Address?
Traditional voice assistants operate in a "walkie-talkie" mode: you speak, you stop, and then the AI responds. Modern "full-duplex" systems, however, are designed to listen and speak simultaneously, handling interruptions, backchannels (like "mhm" or "okay"), and mid-sentence pauses.
Until now, benchmarking these systems has been fragmented. One test might measure how quickly a model stops talking when interrupted, while another tests its ability to remember a grocery list, often using different datasets and protocols. This makes it nearly impossible to compare systems holistically. InteractionBench addresses this by providing a single "harness" that evaluates four critical dimensions of interaction using the exact same audio stimuli for every system.
Methods
The researchers built an evaluation harness that streams pre-recorded user audio to various AI systems in real-time.
Key methodological innovations include:
- Unified Timeline: Both user and system audio are recorded on a single two-track timeline, allowing for millisecond-level precision in measuring turn-taking and interruptions.
- Timeline Reconstruction: For systems like OpenAI’s GPT-Live, which omit silent frames in their stream, the authors developed a method to rebuild the timeline based on arrival timestamps to ensure fair timing measurements.
- Hybrid Grading: The benchmark uses deterministic rules for timing and acoustic analysis (e.g., detecting whispers via harmonics-to-noise ratios) and an LLM judge (Gemini 2.5 Flash) to evaluate the semantic correctness of answers and conversational grounding.
Datasets
The benchmark consists of 221 items per system across 13 distinct scenarios.
- Stimuli: User speech was synthesized using MiMo TTS, including variations in accent, volume (whispering/shouting), and style.
- Scenarios: These include timing tests (pauses, interruptions), spoken tasks (logic puzzles, countdowns), and paralinguistic control.
- Groundedness Probes: The authors generated 10 long-form conversations (22–24 turns each) containing 52 specific "probes" designed to test if the AI can maintain a consistent state, reject false accusations, and flag nonsense questions.
Key Insights and Ideas
- The Full-Duplex Correlation: Among native full-duplex models, there is a strong correlation ($\rho = 0.83$) between timing and task accuracy. Systems that are better at knowing when to speak are generally better at knowing what to say.
- The "Network Delay" Illusion: Timing measured at the client (where the user is) includes network latency. For GPT-Live, the authors found that removing a ~1.5s network delay actually lowered its timing score from 86% to 56%. Without the delay, the model frequently started speaking before the user had finished their sentence.
- The Proactivity Gap: No system successfully passed the "Grammar" scenario, which required the AI to interrupt the user immediately upon hearing a grammatical error. This suggests that current models are still largely reactive rather than truly proactive.
- Whispering is a Rare Skill: Despite the capability being marketed, only Gemini Live whispered reliably (8/10 items). Most other systems responded to whispered requests in a normal voice.
Key Results
The evaluation compared hosted APIs (GPT-Live, GPT-Realtime, Gemini Live), open models (Moshi, PersonaPlex, Freeze-Omni), and a traditional turn-based "cascade" (ASR + LLM + TTS).
- Task Accuracy: The hosted systems and the turn-based cascade were neck-and-neck, passing 68–76% of tasks. Open full-duplex models lagged significantly, with Moshi passing only 6%.
- Groundedness: Hosted systems were far superior at maintaining conversational state, passing 67–88% of probes, compared to just 14–35% for open models.
- Timing: GPT-Live achieved the highest timing score at the client level (86%), while Moshi showed the lowest median latency (190ms) but often spoke over the user inappropriately.
- The Cascade Paradox: The turn-based cascade had the highest task accuracy (76%) but the worst timing for natural interaction, as it cannot handle pauses or interruptions within a turn.
Limitations and Future Work
The authors acknowledge several limitations:
- Synthetic Speech: The use of TTS-generated audio lacks the "messiness" (background noise, stutters, emotional variance) of real human speech.
- Instruction Sensitivity: Different systems required slightly different text prompts to function correctly, which may have influenced their behavior.
- Narrow Scope: The benchmark does not yet measure "naturalness," safety, or long-horizon task completion (e.g., booking a flight).
Future work aims to incorporate real human speech, measure network delay across all hosted providers, and standardize instructions to further level the playing field.
Summary updated