EchoChain: A Full-Duplex Benchmark for State-Update Reasoning Under Interruptions

Smit Nautambhai Modi, Gandharv Mahajan, Marc Wetter, Randall WellesarXiv 2026 · OpenReview

TL;DR

This paper introduces EchoChain, a novel benchmark designed to evaluate how well voice assistants can update their internal state and reasoning when interrupted mid-response. They found that current state-of-the-art models struggle significantly with this, with no system achieving a 50% pass rate, highlighting a critical gap in full-duplex voice interaction capabilities.

What Problem Does This Paper Address?

Human conversation is inherently full-duplex, meaning participants can speak and listen simultaneously. While voice assistants are becoming more sophisticated, their evaluation has largely been confined to half-duplex scenarios, where one turn must complete before the next begins. This overlooks a crucial aspect of real-world interaction: users frequently interrupt assistants mid-sentence to correct information, add constraints, or change their minds. Existing benchmarks don't adequately capture how well these systems can revise their in-progress reasoning and task state when new information arrives during generation. This failure mode is often subtle, with assistants appearing fluent but operating on outdated or incorrect assumptions.

Methods

The authors developed EchoChain, a controlled benchmark that simulates full-duplex interactions with mid-speech interruptions. The framework generates scenario-driven conversations using a planner-simulator loop, ensuring interruptions are contextually grounded. Key to EchoChain is its ability to inject interruptions at a standardized point relative to the assistant's speech onset, allowing for fair cross-model comparisons. User speech and interruption stimuli are pre-synthesized using a Persona Voice Engine to ensure consistent audio input. The benchmark also introduces a diagnostic failure taxonomy with three categories: contextual inertia, interruption amnesia, and objective displacement, to classify specific reasoning failures.

Datasets

EchoChain is a newly introduced benchmark. It generates its own conversational data through a planner-simulator loop, creating scenario-specific dialogues for evaluation.

Key Insights and Ideas

  • Interruptions are a major failure point: The study found that a significant portion of errors (40.22% reduction in failures in a paired half-duplex control) are directly attributable to the interruption event itself, rather than general task difficulty.
  • Current models lack robust state-update reasoning: No evaluated model achieved a pass rate above 50%, indicating a substantial need for improvement in how voice assistants handle mid-generation revisions.
  • Failure taxonomy is crucial: The three identified failure patterns (contextual inertia, interruption amnesia, objective displacement) provide actionable diagnostic signals, helping researchers pinpoint specific weaknesses in models.
  • Full-duplex is more than just overlap: EchoChain moves beyond simply detecting overlap to evaluating the reasoning that occurs after an interruption, a critical distinction for true full-duplex capability.

Key Results

Across four state-of-the-art full-duplex voice models (GPT-realtime-2025-08-28, Gemini Live-2.5-flash-native-audio, Grok Voice Agent, and Amazon Nova Sonic 2), the Mean Pass Rate (MPR) remained below 50%. Grok Voice Agent performed best with an MPR of 48.5%. The failure taxonomy revealed distinct patterns: Gemini Live struggled with interruption amnesia and objective displacement, while Amazon Nova Sonic 2 showed a tendency towards contextual inertia. GPT-realtime had a more balanced distribution of failures.

Limitations and Future Work

The authors acknowledge several limitations:

  • EchoChain currently only evaluates interruptions that deliver new task-relevant information, excluding other forms of full-duplex input like backchanneling or ambient noise.
  • Barge-in timing is fixed for reproducibility, which might not always reflect the most natural interruption points.
  • The evaluation relies on discrete, conversation-specific rubrics, which might miss nuanced failures or cases where severity is ambiguous.
  • The current evaluation focuses on closed-source models and state-update reasoning, not acoustic aspects like response timing or prosody.

Future work could expand to include other forms of full-duplex input, explore more dynamic interruption timing, and incorporate acoustic evaluation.

Summary updated

BibTeX
@misc{modi2026echochain,
  author = {Smit Nautambhai Modi and Gandharv Mahajan and Marc Wetter and Randall Welles},
  title = {{EchoChain: A Full-Duplex Benchmark for State-Update Reasoning Under Interruptions}},
  year = {2026},
  url = {https://openreview.net/forum?id=qpPll9fmKu},
}
Figure