Introduction
Traditional world models have primarily focused on visual perception, effectively leaving agents "deaf" to the rich spatial and temporal cues provided by sound. In the real world, perception is inherently multisensory; audio allows for locating objects beyond the line of sight and understanding the acoustic properties of an environment. This paper introduces the first formal framework for Audio-Visual World Models (AVWM), enabling agents to "imagine" both sight and sound simultaneously.
By formulating environment simulation as a multimodal POMDP, the authors bridge the gap between vision-only prediction and full multisensory reasoning. They introduce a novel architecture, AV-CDiT, and a specialized dataset, AVW-4k, to facilitate the development of agents that can navigate complex 3D spaces using synchronized binaural audio and visual dynamics.
Methods: The AV-CDiT Architecture
The core of the proposed framework is the Audio-Visual Conditional Diffusion Transformer (AV-CDiT). This model is designed to predict future states—comprising images, binaural audio segments, and rewards—conditioned on precise low-level action sequences.
Technical Innovations
- Modality Expert Architecture: To prevent the dominant visual modality from overwhelming auditory signals, the Transformer blocks utilize separate "expert" feed-forward networks for visual and auditory tokens. This architectural choice ensures that the model learns to respect the subtle nuances of binaural audio.
- Stagewise Training Strategy: Optimization is handled in three distinct phases to ensure stability:
- Vision-only training to establish spatial-temporal foundations.
- Audio-only fine-tuning (with frozen vision layers) to isolate auditory reasoning.
- Joint end-to-end fine-tuning for synchronized multimodal fusion.
- Action-Conditioned Diffusion: The model generates the next sensory state based on specific actions like "move forward 0.15m" or "turn left 10°," ensuring the simulated environment responds realistically to agent behavior.
The AVW-4k Dataset
A significant contribution of this work is the AVW-4k dataset, which addresses the lack of physically consistent, action-annotated multimodal data.
- Scale: Includes approximately 30 hours of data across 4,500 trajectories in 76 indoor environments.
- Consistency: Built using Matterport3D and SoundSpaces 2.0, it maintains strict physical AV-consistency, where binaural audio changes realistically as the agent moves through 3D space.
- Precision: Unlike existing datasets like AudioSet or Ego4D, AVW-4k provides the precise action logs necessary for training world models that can be used for planning.
Key Results and Insights
The researchers found that integrating binaural audio significantly enhances spatial reasoning. AV-CDiT achieved state-of-the-art fidelity, outperforming factorized baselines in metrics such as LPIPS (visual similarity) and Fréchet Audio Distance (audio realism).
In embodied navigation tasks, the AVWM-enhanced agent demonstrated superior efficiency. By using the world model as a "mental simulator" to roll out potential futures, the agent reached goals with a reduced number of actions (improving from 332 to 317 actions on average). Ablation studies confirmed that both the "modality experts" and the "stagewise training" are critical; removing them led to a significant decline in auditory quality (higher LSD and FAD scores).
Conclusion and Future Work
AVWMs represent a major step toward more capable embodied AI. While current evaluations rely on synthetic data, the framework establishes the groundwork for multisensory imagination. Future research will focus on scaling these models to real-world datasets and incorporating additional modalities such as touch or depth to further enrich the agent's understanding of the physical world.
Summary updated