Introduction
Recent advancements in 4D generation have enabled the creation of photorealistic, dynamic 3D scenes from monocular videos. However, while visual fidelity has reached impressive heights, the auditory component often remains monaural, failing to provide the spatial immersion required for true AR/VR experiences. Sonic4D addresses this gap by introducing a training-free framework that generates viewpoint-aware spatial audio. By decoupling semantic audio generation from spatial rendering, Sonic4D ensures that sound realistically shifts based on the user's position and the movement of sound sources within a 4D environment.
Methodology
Sonic4D employs a modular, three-stage pipeline that leverages specialized "expert" models to bypass the need for large-scale spatial-audio training data.
1. Dynamic Scene and Monaural Audio Generation
The process begins with a monocular video input. TrajectoryCrafter generates a 4D visual scene represented as dynamic point clouds, while MMAudio produces a high-quality monaural audio track semantically aligned with the video content.
2. 3D Sound-Source Localization and Tracking
To enable physics-based simulation, the system must locate the sound source in 3D space. GroundingGPT, a Multimodal Large Language Model (MLLM), identifies the sound source in 2D frames. These coordinates are back-projected into 3D using depth maps. To ensure a smooth trajectory, DBSCAN clustering is applied to filter tracking noise.
3. Physics-Driven Spatial Audio Synthesis
The final stage uses the estimated 3D source trajectory and the user's virtual camera path to simulate Room Impulse Responses (RIRs) via the Image Source Method (ISM). This transforms the monaural audio into realistic binaural spatial audio, accounting for distance, orientation, and reverberation.
Results and Evaluation
The framework was evaluated using the STARSS23 dataset and a comprehensive user study.
Quantitative Performance
In Sound Event Localization and Detection (SELD) tasks, Sonic4D significantly outperformed the baseline ViSAGe model. It achieved a 4.0% F-score (compared to ViSAGe's 0.4%) and reduced the Direction of Arrival (DOA) angular error from 31.5° to 18.6°. Furthermore, on a 5-point scale for Audio-Visual Spatial Alignment, Sonic4D scored 3.977, nearly double the baseline's 2.418.
Qualitative Analysis
User studies revealed an 89.03% preference rate for Sonic4D over standard monaural audio. As shown in the helicopter example, the generated spatial audio exhibits clear differences between the left (L) and right (R) channels that correspond to the visual position of the moving object, unlike the uniform monaural track.
Discussion and Conclusion
Sonic4D demonstrates that decoupling "what" is heard from "where" it is heard is a highly effective strategy for spatial audio generation. By utilizing MLLMs as visual grounders and classical physics for synthesis, the framework remains training-free and adaptable to future improvements in underlying models.
While currently optimized for single-source scenes, Sonic4D sets a new standard for immersive 4D exploration. Future work will focus on handling complex multi-source environments and extracting more detailed acoustic properties from the 3D scene geometry to further refine reverberation profiles.
Summary updated