TL;DR
Duo v1 introduces the Reciprocal World Model (RWM), a framework that treats intelligence and physical reality as equally fundamental components of a simulated world. By coupling an autoregressive "Intent" module with a causal "Realization" module, Duo enables real-time, full-duplex audiovisual interactions where the AI can perceive, reason, and respond to users—even handling interruptions—without traditional turn-based constraints.
What Problem Does This Paper Address?
Most current "world models" (like Sora or Genie) focus almost exclusively on simulating physical reality—predicting how a scene evolves based on external controls like text prompts or gamepad inputs. However, these worlds often feel "lifeless" because the intelligence resides outside the model.
Furthermore, human interaction is naturally full-duplex: we listen and speak simultaneously, pick up on non-verbal cues, and interrupt one another. Standard AI assistants are typically half-duplex (turn-based), leading to a "stop-and-wait" lag that feels unnatural. Duo aims to bridge this gap by making reasoning and audiovisual generation intrinsic, synchronized parts of a single, continuous streaming system.
Methods
The Duo v1 architecture is built on two pillars:
- Multimodal Intent Module: A pretrained language backbone adapted for causal, chunk-wise interaction. Instead of just understanding instructions, it reasons over the ongoing interaction and makes "participation decisions" at every chunk (e.g., emitting a
<speak>or<wait>action). - Causal Audiovisual Realization Module: A diffusion transformer that jointly generates audio and video latents. It is conditioned on the hidden states of the Intent module, providing semantic guidance for the assistant's behavior.
Key Technical Innovations:
- Causal Chunking: Interaction is divided into 17-frame chunks (~0.7s). Module-specific attention masks ensure that the model only sees the past and current chunk, preventing "future leakage" and enabling real-time response.
- Dual Rollout Distillation: A major challenge in streaming models is "distribution shift"—the difference between clean training data and the noisy, self-generated history the model sees during inference. Duo uses a distillation process where the model is trained on its own autoregressive rollouts to improve stability.
- Unified Timeline: All modalities (user audio/video, assistant text/media) are synchronized on a single timeline using 3D-RoPE (Rotary Positional Embedding) to maintain spatial and temporal relationships.
Datasets
The authors developed a massive, diverse data pipeline to train Duo:
- Synthetic Data: They used LLMs to write "screenplays" for two-person video calls, which were then rendered using a text-to-video (T2V) model. They introduced a "fixed opening anchor" method to prevent visual drift (like darkening skin tones or losing facial detail) over long sequences.
- Recorded Data: Over 3,000 hours of real-world data, including podcasts, synchronous/asynchronous dual-stream videos, and live streams with audience comments.
Key Insights and Ideas
- Intelligence as Physics: The core philosophical shift is treating "Intent" (cognition) as a physical property of the world model, rather than an external input.
- The Power of
<wait>: By explicitly training the model to predict a<wait>token, the system learns when not to speak, which is just as important for natural flow as knowing when to start. - Action-Aware Rebalancing: Since most interaction chunks contain no new speech, the authors had to carefully weight the loss function to prevent the model from collapsing into a state of perpetual silence.
- Visual Grounding: The model isn't just a chatbot; it perceives the user's environment. If a user shows a plant, the "Intent" module incorporates that visual evidence into its reasoning.
Key Results
Duo v1 was evaluated on 30 real-time duplex interaction cases against six binary criteria:
- Overall Success: 93.3% of checks passed.
- Fidelity: The model achieved a 100% pass rate on language adherence, speech intelligibility, and visual identity stability.
- Synchronization: Lip-sync and cross-modal emotional consistency were high, at 90% and 96.7%, respectively.
- Latency: The system achieves a delay to the first visible output of approximately 330–650ms, which is within the threshold for interactive tasks.
Limitations and Future Work
The primary area for improvement is Content Responsiveness, which reached 73.3%. The authors note that in complex scenarios—like a user making a joke or providing intent that only becomes clear when speech and video are combined—the model can still struggle to settle on the perfect reply in real time.
Future work will focus on deeper inference of user meaning at conversational speeds and expanding the model's ability to handle longer, more complex sessions. The team has already signaled that Duo v2 is in development.
Summary updated