CrossLiDAR: Controllable Multi-Sensor LiDAR Sequence Generation

Khanh Nguyen, Thomas Walker, Hichem Telli, Carlos de la Torre-Ortiz, Stefano Fiorini, Salah Eddine Bekhouche, Mohammed-En-Nadhir Zighem, Wassim Swaileh, Tinghuai WangarXiv 2026 · arXiv

TL;DR

CrossLiDAR is a novel generative framework that synthesizes synchronized, controllable LiDAR data from multiple sensors simultaneously. By combining a rectified flow transformer with specialized modules for cross-sensor alignment and temporal consistency, it enables high-fidelity, long-horizon 4D simulations that significantly outperform existing single-sensor baselines.

What Problem Does This Paper Address?

High-fidelity LiDAR simulation is essential for developing and validating autonomous systems, especially for testing dangerous "corner cases." However, current generative models face three major hurdles:

  1. Single-Sensor Constraint: Most models generate data for only one sensor, whereas real autonomous vehicles use heterogeneous suites (e.g., one roof-mounted and four side-mounted sensors) with overlapping fields of view.
  2. Geometric Inconsistency: Naively merging independent sensor outputs leads to misalignments and "ghosting" artifacts.
  3. Temporal Drift: Generating long sequences (4D generation) often results in structural degradation and "fuzzy" geometries over time.

CrossLiDAR tackles these by introducing a unified framework designed for multi-sensor, spatio-temporally consistent, and controllable generation.

Methods

The authors depart from traditional U-Net architectures, opting instead for a Multi-Modal Diffusion Transformer (MM-DiT) backbone. The pipeline operates in three main stages:

  • Latent Compression: A Variational Autoencoder (VAE) compresses high-dimensional range images into a compact latent space.
  • Rectified Flow Generation: The model uses a rectified flow objective, which encourages near-linear transport paths from noise to data, allowing for faster and more accurate sampling.
  • Specialized Modules:
    • Inter-sensor Module: Uses joint attention to model interactions between different sensors at the same timestamp, ensuring they "see" the same world consistently.
  • LiDAR Sequence Module: Captures temporal dependencies across frames using causal masking to prevent information leakage from the future.
  • ControlNet Branch: Provides fine-grained control by conditioning the generation on 3D bounding boxes and Bird’s-Eye-View (BEV) maps.

To maintain long-term stability, the model generates short clips autoregressively. During training, they use a noise-augmented strategy—adding a small amount of noise to the conditioning frames—to make the model robust to its own errors during long rollouts.

Figure 2. Overview of CrossLiDAR Framework. Our framework utilizes a latent rectified-flow [26] formulation to generate synchronized multi-sensor LiDAR sequences. (Left) A frozen VAE encoder first maps range images into a compact latent space zt. (Right) These latents are processed by an MM-DiT [9] backbone, which incorporates two different attention mechanisms: an Inter-sensor Module for cross- sensor alignment and a LiDAR Sequence Module for temporal consistency. The generation is guided by a dedicated ControlNet [70] branch for BEV and 3D layout control. Finally, a frozen VAE decoder transforms the denoised latents back into high-fidelity temporal point clouds.

Datasets

The model was evaluated on two major benchmarks:

  • Waymo Open Dataset: Used specifically for its multi-sensor configuration (Top LiDAR + 4 Side LiDARs).
  • nuScenes: Used to evaluate single-sensor performance on 32-beam LiDAR sweeps.

Because most existing baselines are single-sensor, the authors adapted them using a "naive stacking" strategy to provide a fair comparison in multi-sensor settings.

Key Insights and Ideas

  • Transformers > CNNs for LiDAR: Range images have extreme aspect ratios (e.g., 32x1024). Transformers handle these long-range horizontal dependencies much better than the local receptive fields of standard CNN-based U-Nets.
  • Hardware Decoupling: The Inter-sensor Module allows the model to handle various sensor counts and layouts without needing a custom architecture for every vehicle configuration.
  • Robust Autoregression: The "aha moment" for long-horizon stability was the noise-augmented training. By training the model to handle slightly noisy previous frames, they effectively mitigated the "exposure bias" that usually causes generative models to drift into nonsense over time.

Key Results

CrossLiDAR achieves state-of-the-art (SOTA) results across the board:

  • Waymo (Multi-sensor): Improved Frechet Range Distance (FRD) by 93.0% and reduced Frechet Point Distance (FPD) by 71.7% compared to the best baselines.
  • nuScenes (Single-sensor): Achieved a 57.1% reduction in FPD, showing that the architecture is superior even without the multi-sensor benefits.
  • Temporal Consistency: In 16-frame rollouts on nuScenes, CrossLiDAR reduced temporal error (TTCE) by over 60% compared to previous methods, maintaining sharp object shapes where others became "fuzzy."
Figure 1. CrossLiDAR jointly generates synchronized LiDAR sequences for multi-sensor setups. Top: given multiple sensors mounted around the ego vehicle, CrossLiDAR synthesizes geo- metrically aligned per-sensor views (front/rear/left/right) that can be fused into a consistent ego-frame point cloud, while support- ing structural conditions such as 3D bounding boxes. Bottom: our pipeline operates in a VAE-compressed range-image latent space; a rectified-flow module integrates an Inter-sensor Module for sensor-aware generation and a ControlNet branch for structural conditioning using BEV maps with projected 3D bounding boxes, yielding controllable multi-sensor LiDAR sequences.

Limitations and Future Work

While a major step forward, the authors note several areas for growth:

  • Quantitative Control: There is currently no metric to strictly quantify how well the generated points adhere to the input 3D bounding boxes (foreground fidelity).
  • Downstream Validation: The ultimate test—whether this simulated data can effectively train a 3D object detector or a motion planner—remains for future work.
  • Sensor Adaptability: The current evaluation focuses on fixed configurations; future research could explore how well the model adapts to arbitrary sensor counts and calibrations on the fly.

Summary updated

Abstract

High-fidelity LiDAR simulation remains a bottleneck for scalable development and validation of autonomous systems. Existing generators can synthesize realistic scans, but they are typically constrained to a single sensor configuration, offer limited control over structure and geometry, and drift in long-horizon 4D generation. We present CrossLiDAR, a unified transformer-based framework capable of generating controllable and spatio-temporally consistent multi-sensor LiDAR sequences. Our approach leverages a rectified flow objective with a Multi-Modal Diffusion Transformer (MM-DiT) backbone, within a compact latent space of range images. To support multiple LiDAR sensors, an Inter-sensor Module enables interaction across sensors, while a ControlNet-style branch provides structural conditioning (e.g., BEV priors or 3D bounding boxes). For modeling the temporal consistency, CrossLiDAR generates short $N$-frame sequences and extends them autoregressively to improve long-horizon coherence. Our model significantly outperforms state-of-the-art baselines, achieving an average improvement of 29.56\% in JSD and 15.36\% in MMD evaluated on nuScenes and the Waymo Dataset. CrossLiDAR bridges the gap to modern autonomous-driving simulation by enabling calibration-consistent multi-sensor LiDAR rollouts over long horizons for controllable, scenario-based testing.
BibTeX
@misc{nguyen2026crosslidar,
  author = {Khanh Nguyen and Thomas Walker and Hichem Telli and Carlos de la Torre-Ortiz and Stefano Fiorini and Salah Eddine Bekhouche and Mohammed-En-Nadhir Zighem and Wassim Swaileh and Tinghuai Wang},
  title = {{CrossLiDAR: Controllable Multi-Sensor LiDAR Sequence Generation}},
  year = {2026},
  eprint = {2611.11131},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11131},
}
Figure