TL;DR
CrossLiDAR is a novel generative framework that synthesizes synchronized, controllable LiDAR data from multiple sensors simultaneously. By combining a rectified flow transformer with specialized modules for cross-sensor alignment and temporal consistency, it enables high-fidelity, long-horizon 4D simulations that significantly outperform existing single-sensor baselines.
What Problem Does This Paper Address?
High-fidelity LiDAR simulation is essential for developing and validating autonomous systems, especially for testing dangerous "corner cases." However, current generative models face three major hurdles:
- Single-Sensor Constraint: Most models generate data for only one sensor, whereas real autonomous vehicles use heterogeneous suites (e.g., one roof-mounted and four side-mounted sensors) with overlapping fields of view.
- Geometric Inconsistency: Naively merging independent sensor outputs leads to misalignments and "ghosting" artifacts.
- Temporal Drift: Generating long sequences (4D generation) often results in structural degradation and "fuzzy" geometries over time.
CrossLiDAR tackles these by introducing a unified framework designed for multi-sensor, spatio-temporally consistent, and controllable generation.
Methods
The authors depart from traditional U-Net architectures, opting instead for a Multi-Modal Diffusion Transformer (MM-DiT) backbone. The pipeline operates in three main stages:
- Latent Compression: A Variational Autoencoder (VAE) compresses high-dimensional range images into a compact latent space.
- Rectified Flow Generation: The model uses a rectified flow objective, which encourages near-linear transport paths from noise to data, allowing for faster and more accurate sampling.
- Specialized Modules:
- Inter-sensor Module: Uses joint attention to model interactions between different sensors at the same timestamp, ensuring they "see" the same world consistently.
- LiDAR Sequence Module: Captures temporal dependencies across frames using causal masking to prevent information leakage from the future.
- ControlNet Branch: Provides fine-grained control by conditioning the generation on 3D bounding boxes and Bird’s-Eye-View (BEV) maps.
To maintain long-term stability, the model generates short clips autoregressively. During training, they use a noise-augmented strategy—adding a small amount of noise to the conditioning frames—to make the model robust to its own errors during long rollouts.
Datasets
The model was evaluated on two major benchmarks:
- Waymo Open Dataset: Used specifically for its multi-sensor configuration (Top LiDAR + 4 Side LiDARs).
- nuScenes: Used to evaluate single-sensor performance on 32-beam LiDAR sweeps.
Because most existing baselines are single-sensor, the authors adapted them using a "naive stacking" strategy to provide a fair comparison in multi-sensor settings.
Key Insights and Ideas
- Transformers > CNNs for LiDAR: Range images have extreme aspect ratios (e.g., 32x1024). Transformers handle these long-range horizontal dependencies much better than the local receptive fields of standard CNN-based U-Nets.
- Hardware Decoupling: The Inter-sensor Module allows the model to handle various sensor counts and layouts without needing a custom architecture for every vehicle configuration.
- Robust Autoregression: The "aha moment" for long-horizon stability was the noise-augmented training. By training the model to handle slightly noisy previous frames, they effectively mitigated the "exposure bias" that usually causes generative models to drift into nonsense over time.
Key Results
CrossLiDAR achieves state-of-the-art (SOTA) results across the board:
- Waymo (Multi-sensor): Improved Frechet Range Distance (FRD) by 93.0% and reduced Frechet Point Distance (FPD) by 71.7% compared to the best baselines.
- nuScenes (Single-sensor): Achieved a 57.1% reduction in FPD, showing that the architecture is superior even without the multi-sensor benefits.
- Temporal Consistency: In 16-frame rollouts on nuScenes, CrossLiDAR reduced temporal error (TTCE) by over 60% compared to previous methods, maintaining sharp object shapes where others became "fuzzy."
Limitations and Future Work
While a major step forward, the authors note several areas for growth:
- Quantitative Control: There is currently no metric to strictly quantify how well the generated points adhere to the input 3D bounding boxes (foreground fidelity).
- Downstream Validation: The ultimate test—whether this simulated data can effectively train a 3D object detector or a motion planner—remains for future work.
- Sensor Adaptability: The current evaluation focuses on fixed configurations; future research could explore how well the model adapts to arbitrary sensor counts and calibrations on the fly.
Summary updated