Audio-Visual World Models: Towards Multisensory Imagination in Sight and Sound

Jiahua Wang, Shannan Yan, Leqi Zheng, Jialong Wu, Yaoxin Mao

Introduction

Traditional world models have primarily focused on visual perception, effectively leaving agents "deaf" to the rich spatial and temporal cues provided by sound. In the real world, perception is inherently multisensory; audio allows for locating objects beyond the line of sight and understanding the acoustic properties of an environment. This paper introduces the first formal framework for Audio-Visual World Models (AVWM), enabling agents to "imagine" both sight and sound simultaneously.

Conceptual comparison of Real World, Visual WMs, and the proposed AVWMs.

By formulating environment simulation as a multimodal POMDP, the authors bridge the gap between vision-only prediction and full multisensory reasoning. They introduce a novel architecture, AV-CDiT, and a specialized dataset, AVW-4k, to facilitate the development of agents that can navigate complex 3D spaces using synchronized binaural audio and visual dynamics.

Methods: The AV-CDiT Architecture

The core of the proposed framework is the Audio-Visual Conditional Diffusion Transformer (AV-CDiT). This model is designed to predict future states—comprising images, binaural audio segments, and rewards—conditioned on precise low-level action sequences.

The AV-CDiT Architecture featuring modality experts and action-conditioned diffusion.

Technical Innovations

  • Modality Expert Architecture: To prevent the dominant visual modality from overwhelming auditory signals, the Transformer blocks utilize separate "expert" feed-forward networks for visual and auditory tokens. This architectural choice ensures that the model learns to respect the subtle nuances of binaural audio.
  • Stagewise Training Strategy: Optimization is handled in three distinct phases to ensure stability:
  • Vision-only training to establish spatial-temporal foundations.
  • Audio-only fine-tuning (with frozen vision layers) to isolate auditory reasoning.
  • Joint end-to-end fine-tuning for synchronized multimodal fusion.
  • Action-Conditioned Diffusion: The model generates the next sensory state based on specific actions like "move forward 0.15m" or "turn left 10°," ensuring the simulated environment responds realistically to agent behavior.

The AVW-4k Dataset

A significant contribution of this work is the AVW-4k dataset, which addresses the lack of physically consistent, action-annotated multimodal data.

  • Scale: Includes approximately 30 hours of data across 4,500 trajectories in 76 indoor environments.
  • Consistency: Built using Matterport3D and SoundSpaces 2.0, it maintains strict physical AV-consistency, where binaural audio changes realistically as the agent moves through 3D space.
  • Precision: Unlike existing datasets like AudioSet or Ego4D, AVW-4k provides the precise action logs necessary for training world models that can be used for planning.

Key Results and Insights

The researchers found that integrating binaural audio significantly enhances spatial reasoning. AV-CDiT achieved state-of-the-art fidelity, outperforming factorized baselines in metrics such as LPIPS (visual similarity) and Fréchet Audio Distance (audio realism).

In embodied navigation tasks, the AVWM-enhanced agent demonstrated superior efficiency. By using the world model as a "mental simulator" to roll out potential futures, the agent reached goals with a reduced number of actions (improving from 332 to 317 actions on average). Ablation studies confirmed that both the "modality experts" and the "stagewise training" are critical; removing them led to a significant decline in auditory quality (higher LSD and FAD scores).

Conclusion and Future Work

AVWMs represent a major step toward more capable embodied AI. While current evaluations rely on synthetic data, the framework establishes the groundwork for multisensory imagination. Future research will focus on scaling these models to real-world datasets and incorporating additional modalities such as touch or depth to further enrich the agent's understanding of the physical world.

Summary updated

Abstract

World models simulate environmental dynamics to enable embodied agents to plan and reason about future states. While real-world perception is inherently multimodal, existing approaches focus primarily on visual observations, leaving crucial spatial and temporal acoustic cues underexplored. In this work, we present a unified formulation of Audio-Visual World Models (AVWM), casting multimodal environment simulation under action control as a partially observable Markov decision process with synchronized audio-visual observations. As a foundational benchmark, we construct AVW-4k, comprising 30 hours of action-annotated binaural audio-visual trajectories across 76 indoor environments. To capture these physically grounded multisensory dynamics, we propose AV-CDiT (Audio-Visual Conditional Diffusion Transformer), featuring a novel modality expert architecture that balances visual and auditory learning, optimized via a three-stage training strategy. Extensive experiments demonstrate that AV-CDiT achieves high-fidelity prediction across both visual and auditory modalities. Furthermore, we validate its practical utility in embodied navigation, showing that AVWM significantly enhances a pretrained agent in continuous audio-visual navigation tasks.
BibTeX
@misc{wang2025audio,
  author = {Jiahua Wang and Shannan Yan and Leqi Zheng and Jialong Wu and Yaoxin Mao},
  title = {{Audio-Visual World Models: Towards Multisensory Imagination in Sight and Sound}},
  year = {2025},
  eprint = {2512.00883},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.00883},
}
Figure