R3DPA: Leveraging 3D Representation Alignment and RGB Pretrained Priors for LiDAR Scene Generation

Nicolas Sereyjol-Garros, Ellington Kirby, Victor Besnier, Nermin Samet

Abstract

LiDAR scene synthesis is an emerging solution to scarcity in 3D data for robotic tasks such as autonomous driving. Recent approaches employ diffusion or flow matching models to generate realistic scenes, but 3D data remains limited compared to RGB datasets with millions of samples. We introduce R3DPA, the first LiDAR scene generation method to unlock image-pretrained priors for LiDAR point clouds, and leverage self-supervised 3D representations for state-of-the-art results. Specifically, we (i) align intermediate features of our generative model with self-supervised 3D features, which substantially improves generation quality; (ii) transfer knowledge from large-scale image-pretrained generative models to LiDAR generation, mitigating limited LiDAR datasets; and (iii) enable point cloud control at inference for object inpainting and scene mixing with solely an unconditional model. On the KITTI-360 benchmark R3DPA achieves state of the art performance.

Introduction

LiDAR sensors provide the precise 3D spatial data necessary for safe autonomous navigation. However, the development of generative models for LiDAR is hindered by the high cost of data collection and annotation compared to the abundance of 2D image data. Existing LiDAR generation methods often struggle with data scarcity, leading to limited capture of complex scene distributions. Furthermore, there is a significant representation gap between 2D "foundation models" and 3D point clouds, preventing the effective use of massive image-pretrained priors.

R3DPA (Representation Alignment with 3D Priors and Alignment) addresses these challenges by being the first method to successfully repurpose generative models pretrained on millions of natural images for high-fidelity 3D LiDAR point cloud generation.

Methodology

The R3DPA framework utilizes a two-stage pipeline designed to bridge the gap between 2D image priors and 3D structural requirements.

Comparison of standard LiDAR generative training vs. R3DPA's two-stage alignment approach.

1. VAE Alignment for RGB Priors

Instead of training from scratch, R3DPA initializes a Flow Matching (FM) model with weights from a model pretrained on ImageNet. To bridge the domain gap, a Variational Autoencoder (VAE) is trained while the FM model remains frozen. An alignment loss is employed to force the VAE's latent space to become compatible with the FM model’s pretrained image knowledge, effectively creating a "bridge" between the two domains.

2. End-to-End Training with 3D Supervision

In the second stage, the FM model is unfrozen, and the entire system is trained jointly. A crucial innovation is the supervision of the generative model's internal features using a pretrained 3D backbone (ScaLR). This ensures the model learns structurally sound 3D geometry rather than just mimicking the appearance of 2D projections.

The R3DPA training pipeline, illustrating the integration of the 3D Pretrained Backbone and the Flow Matching Model with specific alignment and denoising losses.

3. Equirectangular Projection and Control

To leverage 2D architectures, unstructured point clouds are projected into 2D "range images," where pixel values represent depth. Because the model is trained to align with specific features, it enables zero-shot control at inference time. By backpropagating the alignment loss, the model can perform tasks like object inpainting or scene mixing without additional training.

Results and Discussion

R3DPA was evaluated on the KITTI-360 benchmark, which contains approximately 80,000 LiDAR scans. The model achieves state-of-the-art performance, notably improving the Fréchet Localization Distance (FLD) and achieving a 17% improvement in point-cloud-based metrics (FSVD/FPVD) over previous best-performing models like R2Flow and LiDM.

The use of a transformer backbone (SiT-B/2) also demonstrates efficiency, utilizing roughly half the parameters of standard UNet architectures (130M vs 266M) while producing sharper object boundaries and more realistic scan patterns compared to previous geometrically inconsistent generations.

Conclusion

R3DPA demonstrates that image-pretrained priors can be effectively transferred to the 3D LiDAR domain through strategic latent space alignment and 3D feature guidance. While the current approach relies on equirectangular projections which may lose fine details, the significant performance gains suggest a promising path for generating high-quality synthetic data to train future perception models. Future work will explore more diverse datasets and the impact of synthetic data on downstream 3D detection tasks.

Summary updated

Abstract

LiDAR scene synthesis is an emerging solution to scarcity in 3D data for robotic tasks such as autonomous driving. Recent approaches employ diffusion or flow matching models to generate realistic scenes, but 3D data remains limited compared to RGB datasets with millions of samples. We introduce R3DPA, the first LiDAR scene generation method to unlock image-pretrained priors for LiDAR point clouds, and leverage self-supervised 3D representations for state-of-the-art results. Specifically, we (i) align intermediate features of our generative model with self-supervised 3D features, which substantially improves generation quality; (ii) transfer knowledge from large-scale image-pretrained generative models to LiDAR generation, mitigating limited LiDAR datasets; and (iii) enable point cloud control at inference for object inpainting and scene mixing with solely an unconditional model. On the KITTI-360 benchmark R3DPA achieves state of the art performance. Code and pretrained models are available at https://github.com/valeoai/R3DPA.
BibTeX
@misc{sereyjolgarros2026r,
  author = {Nicolas Sereyjol-Garros and Ellington Kirby and Victor Besnier and Nermin Samet},
  title = {{R3DPA: Leveraging 3D Representation Alignment and RGB Pretrained Priors for LiDAR Scene Generation}},
  year = {2026},
  eprint = {2601.07692},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.07692},
}
Figure