Sonic4D: Spatial Audio Generation for Immersive 4D Scene Exploration

Siyi Xie, Hanxin Zhu, Tianyu He, Xin Li, Zhibo Chen

Introduction

Recent advancements in 4D generation have enabled the creation of photorealistic, dynamic 3D scenes from monocular videos. However, while visual fidelity has reached impressive heights, the auditory component often remains monaural, failing to provide the spatial immersion required for true AR/VR experiences. Sonic4D addresses this gap by introducing a training-free framework that generates viewpoint-aware spatial audio. By decoupling semantic audio generation from spatial rendering, Sonic4D ensures that sound realistically shifts based on the user's position and the movement of sound sources within a 4D environment.

Methodology

Sonic4D employs a modular, three-stage pipeline that leverages specialized "expert" models to bypass the need for large-scale spatial-audio training data.

The three-stage Sonic4D pipeline: (I) Dynamic scene and monaural audio generation, (II) 3D sound-source localization, and (III) Physics-driven spatial audio synthesis.

1. Dynamic Scene and Monaural Audio Generation

The process begins with a monocular video input. TrajectoryCrafter generates a 4D visual scene represented as dynamic point clouds, while MMAudio produces a high-quality monaural audio track semantically aligned with the video content.

2. 3D Sound-Source Localization and Tracking

To enable physics-based simulation, the system must locate the sound source in 3D space. GroundingGPT, a Multimodal Large Language Model (MLLM), identifies the sound source in 2D frames. These coordinates are back-projected into 3D using depth maps. To ensure a smooth trajectory, DBSCAN clustering is applied to filter tracking noise.

3. Physics-Driven Spatial Audio Synthesis

The final stage uses the estimated 3D source trajectory and the user's virtual camera path to simulate Room Impulse Responses (RIRs) via the Image Source Method (ISM). This transforms the monaural audio into realistic binaural spatial audio, accounting for distance, orientation, and reverberation.

Results and Evaluation

The framework was evaluated using the STARSS23 dataset and a comprehensive user study.

Quantitative Performance

In Sound Event Localization and Detection (SELD) tasks, Sonic4D significantly outperformed the baseline ViSAGe model. It achieved a 4.0% F-score (compared to ViSAGe's 0.4%) and reduced the Direction of Arrival (DOA) angular error from 31.5° to 18.6°. Furthermore, on a 5-point scale for Audio-Visual Spatial Alignment, Sonic4D scored 3.977, nearly double the baseline's 2.418.

Qualitative Analysis

User studies revealed an 89.03% preference rate for Sonic4D over standard monaural audio. As shown in the helicopter example, the generated spatial audio exhibits clear differences between the left (L) and right (R) channels that correspond to the visual position of the moving object, unlike the uniform monaural track.

Qualitative comparison between monaural audio and Sonic4D's spatial audio for a moving helicopter source.

Discussion and Conclusion

Sonic4D demonstrates that decoupling "what" is heard from "where" it is heard is a highly effective strategy for spatial audio generation. By utilizing MLLMs as visual grounders and classical physics for synthesis, the framework remains training-free and adaptable to future improvements in underlying models.

While currently optimized for single-source scenes, Sonic4D sets a new standard for immersive 4D exploration. Future work will focus on handling complex multi-source environments and extracting more detailed acoustic properties from the 3D scene geometry to further refine reverberation profiles.

Summary updated

Abstract

Recent advancements in 4D generation have demonstrated its remarkable capability in synthesizing photorealistic renderings of dynamic 3D scenes. However, despite achieving impressive visual performance, almost all existing methods overlook the generation of spatial audio aligned with the corresponding 4D scenes, posing a significant limitation to truly immersive audiovisual experiences. To mitigate this issue, we propose Sonic4D, a novel framework that enables spatial audio generation for immersive exploration of 4D scenes. Specifically, our method is composed of three stages: 1) To capture both the dynamic visual content and raw auditory information from a monocular video, we first employ pre-trained expert models to generate the 4D scene and its corresponding monaural audio. 2) Subsequently, to transform the monaural audio into spatial audio, we localize and track the sound sources within the 4D scene, where their 3D spatial coordinates at different timestamps are estimated via a pixel-level visual grounding strategy. 3) Based on the estimated sound source locations, we further synthesize plausible spatial audio that varies across different viewpoints and timestamps using physics-based simulation. Extensive experiments have demonstrated that our proposed method generates realistic spatial audio consistent with the synthesized 4D scene in a training-free manner, significantly enhancing the immersive experience for users. Generated audio and video examples are available at https://x-drunker.github.io/Sonic4D-project-page.
BibTeX
@inproceedings{xie2026sonic,
  author = {Siyi Xie and Hanxin Zhu and Tianyu He and Xin Li and Zhibo Chen},
  title = {{Sonic4D: Spatial Audio Generation for Immersive 4D Scene Exploration}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i13.38087},
}
Figure