READ: Large-Scale Neural Scene Rendering for Autonomous Driving

Zhuopeng Li, Lu Li, Zeyu Ma, Ping Zhang, Junbo Chen, Jianke ZhuarXiv 2022 · DOI · arXiv

Abstract

READ (Autonomous Driving scene Render) is a novel neural rendering framework designed to synthesize photo-realistic, large-scale driving environments from sparse point clouds. By employing a multi-scale network architecture and efficient sampling strategies, READ enables the creation, editing, and stitching of extensive driving scenarios on consumer-grade hardware, effectively addressing the "reality gap" in autonomous driving simulation.

Introduction

Developing robust AI for autonomous driving requires vast amounts of diverse training data, including rare "edge cases" like accidents. Traditional simulators often lack realism, leading to a reality gap where models trained in simulation fail in real-world transitions. Existing neural rendering approaches like Image-to-Image GANs often lack 3D consistency, while Neural Radiance Fields (NeRFs) struggle with the scale and computational demands of kilometer-long city streets. Point-based methods, while promising, frequently suffer from artifacts such as "holes" or color bleeding due to sparse input data.

READ supports novel view synthesis, panorama generation, and interactive scene editing.

READ addresses these challenges by providing a scalable framework for high-fidelity scene synthesis and manipulation, as shown in Figure 1.

Methodology

The READ framework centers on the $\omega-net$ architecture, a multi-scale rendering network designed to handle sparse and noisy point cloud data.

The READ pipeline: from source images and point clouds to neural rendering via multi-scale descriptors.

Key Innovations

  1. $\omega-net$ and Gate Module: The network utilizes a Gate Module to identify valid neural descriptors and filter out noise inherent in 3D reconstructions. It fuses features across multiple scales, using low-resolution features to fill gaps in sparse regions and high-resolution features to provide sharp details.
  2. Efficient Sampling: To manage large-scale data on standard GPUs (such as an NVIDIA RTX 2070), READ employs Monte Carlo sampling to focus training on complex textures and Patch sampling to reduce memory overhead.
  3. Scene Manipulation: By associating neural descriptors with specific 3D points, the system allows for object-level editing (e.g., moving vehicles) and the stitching of disparate scene segments into continuous, diverse driving routes.

Experiments and Results

The researchers evaluated READ on the KITTI and Brno Urban datasets. READ significantly outperformed baseline models such as NPBG, ADOP, and DAGAN across all major visual quality metrics.

  • Visual Fidelity: READ achieved the lowest LPIPS and perceptual loss scores, indicating that synthesized images are highly realistic. It maintained higher PSNR and SSIM, resulting in sharper textures on buildings and road signs.
  • Computational Efficiency: The model is highly accessible, requiring only approximately two days of training on a standard PC with two RTX 2070 GPUs.
  • Flexibility: The system successfully demonstrated the ability to "stitch" different residential scenes and move dynamic objects like cars to create entirely new traffic scenarios.

Conclusion

READ provides a scalable and accessible solution for large-scale neural scene rendering in autonomous driving contexts. While performance remains dependent on point cloud accuracy and can exhibit blur in extremely sparse regions, the framework represents a significant step toward closing the reality gap. Future work involving higher-precision LiDAR data may further enhance reconstruction realism and reduce artifacts.

Summary updated

Abstract

Synthesizing free-view photo-realistic images is an important task in multimedia. With the development of advanced driver assistance systems~(ADAS) and their applications in autonomous vehicles, experimenting with different scenarios becomes a challenge. Although the photo-realistic street scenes can be synthesized by image-to-image translation methods, which cannot produce coherent scenes due to the lack of 3D information. In this paper, a large-scale neural rendering method is proposed to synthesize the autonomous driving scene~(READ), which makes it possible to synthesize large-scale driving scenarios on a PC through a variety of sampling schemes. In order to represent driving scenarios, we propose an {\omega} rendering network to learn neural descriptors from sparse point clouds. Our model can not only synthesize realistic driving scenes but also stitch and edit driving scenes. Experiments show that our model performs well in large-scale driving scenarios.
BibTeX
@misc{li2022read,
  author = {Zhuopeng Li and Lu Li and Zeyu Ma and Ping Zhang and Junbo Chen and Jianke Zhu},
  title = {{READ: Large-Scale Neural Scene Rendering for Autonomous Driving}},
  year = {2022},
  eprint = {2205.05509},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2205.05509},
}
Figure