Introduction
Generating high-fidelity, annotated training data is a cornerstone of autonomous driving (AD) development. However, current generative models struggle with two primary challenges: modality silos and the representation gap. Most existing models focus on a single output modality, such as RGB video or LiDAR, which prevents the creation of the synchronized, multi-sensor datasets necessary for robust perception systems. Furthermore, mapping directly from a coarse 2D Bird’s-Eye View (BEV) layout to complex 3D sensor data is inherently difficult due to the lack of geometric and semantic detail in 2D maps.
The UniScene Framework
UniScene addresses these hurdles by introducing 3D semantic occupancy as an intermediate "meta-representation." This structural skeleton is richer than a 2D map but simpler than raw sensor data, serving as a perfect anchor for multi-modal generation.
The framework employs a two-stage hierarchical architecture:
- Controllable Occupancy Generation: A Variational Autoencoder (VAE) compresses 3D occupancy into a continuous latent space, preserving fine spatial details. A Diffusion Transformer (DiT) then generates the occupancy sequence from BEV layouts, ensuring temporal consistency.
- Occupancy-Centric Sensor Generation: The generated occupancy is "fleshed out" into video and LiDAR. For video, occupancy voxels are converted into 3D Gaussian primitives and rendered into 2D maps to guide a Video Diffusion model. For LiDAR, the occupancy guides a ray-based sampling strategy, where a Sparse UNet predicts Signed Distance Functions (SDF) and intensities.
Key Insights: Occupancy as a Universal Language
The core innovation of UniScene is the use of 3D occupancy as a "universal language" for sensor simulation. By providing strict geometric constraints for LiDAR and semantic labels for video, UniScene ensures cross-modal alignment—for instance, ensuring a vehicle's position in a video frame perfectly matches its detection in the LiDAR point cloud. Additionally, the use of Gaussian Splatting bridges the gap between 3D structures and 2D video efficiently, leveraging pre-trained 2D generators while maintaining 3D structural integrity.
Experimental Results
Evaluated on the NuScenes benchmark, UniScene demonstrates significant performance gains over specialized state-of-the-art (SOTA) models. The model's efficiency is particularly noteworthy in LiDAR generation, where it utilizes occupancy to avoid unnecessary computations in empty space.
- Occupancy Generation: Reduced Maximum Mean Discrepancy (MMD) by 71.1% compared to OccWorld.
- Video Generation: Achieved a 41.4% lower Frechet Video Distance (FVD) than models like Vista and MagicDrive.
- LiDAR Generation: Improved MMD by 31.6% over LiDARDM while drastically reducing inference time from 45.12s to 0.25s.
Beyond metrics, data generated by UniScene has been shown to improve the performance of real-world perception models in 3D object detection and BEV segmentation when used as training augmentation.
Conclusion
UniScene represents a major step toward unified world simulators for autonomous driving. By decomposing the generation process into a hierarchical workflow centered on semantic occupancy, it achieves unprecedented consistency and quality across multiple modalities. Future work will focus on reducing computational overhead and integrating closed-loop actor models to enable real-time interactive simulations.
Summary updated