Duo: A Reciprocal World Model for Full-Duplex Interaction

Hengyuan Cao, Biao Gong, Guoshan Lu, Qiyuan Zhang, Lin Liu, Yiding Tian, Yilei Jiang, Yuxin Ma, Chengzhang Wu, Kun Chen, Yiqian Xie, Ting Liu, Xin Zheng, Xijun Gu, Tianren Ma, Jiancheng Pan, Chi Zhang, Xuzhe Zheng, Siyi Li, Zhenzhong LanarXiv 2026 · arXiv

TL;DR

Duo v1 introduces the Reciprocal World Model (RWM), a framework that treats intelligence and physical reality as equally fundamental components of a simulated world. By coupling an autoregressive "Intent" module with a causal "Realization" module, Duo enables real-time, full-duplex audiovisual interactions where the AI can perceive, reason, and respond to users—even handling interruptions—without traditional turn-based constraints.

What Problem Does This Paper Address?

Most current "world models" (like Sora or Genie) focus almost exclusively on simulating physical reality—predicting how a scene evolves based on external controls like text prompts or gamepad inputs. However, these worlds often feel "lifeless" because the intelligence resides outside the model.

Furthermore, human interaction is naturally full-duplex: we listen and speak simultaneously, pick up on non-verbal cues, and interrupt one another. Standard AI assistants are typically half-duplex (turn-based), leading to a "stop-and-wait" lag that feels unnatural. Duo aims to bridge this gap by making reasoning and audiovisual generation intrinsic, synchronized parts of a single, continuous streaming system.

Methods

The Duo v1 architecture is built on two pillars:

  1. Multimodal Intent Module: A pretrained language backbone adapted for causal, chunk-wise interaction. Instead of just understanding instructions, it reasons over the ongoing interaction and makes "participation decisions" at every chunk (e.g., emitting a <speak> or <wait> action).
  2. Causal Audiovisual Realization Module: A diffusion transformer that jointly generates audio and video latents. It is conditioned on the hidden states of the Intent module, providing semantic guidance for the assistant's behavior.
The Duo v1 framework showing the interplay between Intent and Realization modules across temporal chunks.

Key Technical Innovations:

  • Causal Chunking: Interaction is divided into 17-frame chunks (~0.7s). Module-specific attention masks ensure that the model only sees the past and current chunk, preventing "future leakage" and enabling real-time response.
  • Dual Rollout Distillation: A major challenge in streaming models is "distribution shift"—the difference between clean training data and the noisy, self-generated history the model sees during inference. Duo uses a distillation process where the model is trained on its own autoregressive rollouts to improve stability.
  • Unified Timeline: All modalities (user audio/video, assistant text/media) are synchronized on a single timeline using 3D-RoPE (Rotary Positional Embedding) to maintain spatial and temporal relationships.

Datasets

The authors developed a massive, diverse data pipeline to train Duo:

  • Synthetic Data: They used LLMs to write "screenplays" for two-person video calls, which were then rendered using a text-to-video (T2V) model. They introduced a "fixed opening anchor" method to prevent visual drift (like darkening skin tones or losing facial detail) over long sequences.
  • Recorded Data: Over 3,000 hours of real-world data, including podcasts, synchronous/asynchronous dual-stream videos, and live streams with audience comments.
ASR-aligned chunk construction and causal supervision on a shared timeline.

Key Insights and Ideas

  • Intelligence as Physics: The core philosophical shift is treating "Intent" (cognition) as a physical property of the world model, rather than an external input.
  • The Power of <wait>: By explicitly training the model to predict a <wait> token, the system learns when not to speak, which is just as important for natural flow as knowing when to start.
  • Action-Aware Rebalancing: Since most interaction chunks contain no new speech, the authors had to carefully weight the loss function to prevent the model from collapsing into a state of perpetual silence.
  • Visual Grounding: The model isn't just a chatbot; it perceives the user's environment. If a user shows a plant, the "Intent" module incorporates that visual evidence into its reasoning.

Key Results

Duo v1 was evaluated on 30 real-time duplex interaction cases against six binary criteria:

  • Overall Success: 93.3% of checks passed.
  • Fidelity: The model achieved a 100% pass rate on language adherence, speech intelligibility, and visual identity stability.
  • Synchronization: Lip-sync and cross-modal emotional consistency were high, at 90% and 96.7%, respectively.
  • Latency: The system achieves a delay to the first visible output of approximately 330–650ms, which is within the threshold for interactive tasks.

Limitations and Future Work

The primary area for improvement is Content Responsiveness, which reached 73.3%. The authors note that in complex scenarios—like a user making a joke or providing intent that only becomes clear when speech and video are combined—the model can still struggle to settle on the perfect reply in real time.

Future work will focus on deeper inference of user meaning at conversational speeds and expanding the model's ability to handle longer, more complex sessions. The team has already signaled that Duo v2 is in development.

Summary updated

Abstract

In this report, we introduce the Reciprocal World Model (RWM) and present Duo v1, a full-duplex world model that continuously perceives audiovisual inputs while responding to users in real time. The RWM treats intelligence and physical reality as equally fundamental to what constitutes a world. Existing world models largely focus on simulating the latter, leaving reasoning and decision-making as external controls rather than intrinsic parts of the world itself. In RWM, we identify two equally fundamental capabilities: Intent, which provides the cognition underlying world interactions, and Realization, which generates coherent audio-visual behavior. To realize this formulation, Duo v1 introduces three key technical designs: (1) Leveraging autoregressive modeling for continuous interaction reasoning and participation decisions, rather than restricting it to instruction understanding for diffusion models. (2) Constructing chunk-specific conditions and synchronize tokens across Intent and Realization on a unified timeline. Module-specific attention masks enforce causal dependencies both within and across chunks, preventing access to future inputs. (3) Mitigating distribution shift in both modules through dual rollout distillation, training on autoregressively generated Intent tokens and synthesized audiovisual histories from Realization. Both modules continuously perceive real-time audio, video, and text, enabling full-duplex interaction with spontaneous conversations and interruptions. Beyond conversation, users can share real objects, reshape virtual scenes, guide assistants, or play audiovisual games without mouse or keyboard controls. The release of Duo v1 makes our first step, with the next version coming soon. We look forward to broader exploration of this emerging field.
BibTeX
@misc{cao2026duo,
  author = {Hengyuan Cao and Biao Gong and Guoshan Lu and Qiyuan Zhang and Lin Liu and Yiding Tian and Yilei Jiang and Yuxin Ma and Chengzhang Wu and Kun Chen and Yiqian Xie and Ting Liu and Xin Zheng and Xijun Gu and Tianren Ma and Jiancheng Pan and Chi Zhang and Xuzhe Zheng and Siyi Li and Zhenzhong Lan},
  title = {{Duo: A Reciprocal World Model for Full-Duplex Interaction}},
  year = {2026},
  eprint = {2611.15131},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.15131},
}
Figure