MapNeRF: Incorporating Map Priors into Neural Radiance Fields for Driving View Simulation

Chenming Wu, Jiadai Sun, Zhelun Shen, Liangjun Zhang

TL;DR

This paper introduces MapNeRF, a novel approach that leverages High-Definition (HD) map priors to improve the synthesis of driving views, particularly for scenarios where the camera deviates from its original trajectory. By incorporating semantic map information with an uncertainty-aware framework, MapNeRF achieves better semantic consistency and rendering quality in out-of-trajectory view extrapolation, crucial for robust autonomous driving simulation.

What Problem Does This Paper Address?

Autonomous driving (AD) systems require extensive simulation for training and validation. Neural Radiance Fields (NeRFs) have shown great promise in generating photorealistic novel views for these simulations. However, existing NeRF methods struggle when synthesizing views that are outside the original camera trajectory (view extrapolation). This is particularly problematic in AD simulations where vehicles might perform lane changes or other maneuvers that lead to such deviated views. When the camera deviates, the synthesized views often lose semantic consistency, meaning road elements or other scene features might appear distorted or incorrect. This paper aims to bridge this gap by enabling NeRFs to generate semantically consistent and high-quality views even when extrapolating beyond the training trajectory, using readily available map data.

Methods

MapNeRF builds upon the NeRF framework but introduces two key novelties:

  • Ground Density Supervision: The method utilizes the ground height information from HD maps to guide the density field of the NeRF. This is done by defining a probabilistic model for ray termination, distinguishing between uncertain regions (where obstacles might exist), ground regions, and certain regions. The supervision is applied only to the ground and certain regions, using a KL divergence loss and a distance-based weighting to ensure robustness.
  • Multi-view Consistency Supervision with Uncertainty: To maintain semantic consistency across views, MapNeRF employs a multi-view consistency loss. This loss is applied to rays cast from extrapolated views, warped to nearby training views. Crucially, the supervision is weighted by an uncertainty function that measures the confidence of a ray terminating on the ground surface. This allows the model to learn from these extrapolated views without being overly constrained by potentially noisy map data.
  • Uncertainty Tempering: To prevent the training from collapsing due to the inherent uncertainty in map priors and the extrapolation task, MapNeRF introduces an "uncertainty tempering" strategy. This strategy gradually increases the uncertainty parameter over training, allowing the NeRF more freedom to explore and adapt, especially in the early stages of training.

These map-based supervisions are combined with the standard photometric reconstruction loss (Lrgb) to train the final MapNeRF model.

Datasets

The authors train and evaluate MapNeRF on the Argoverse2 dataset. This dataset is specifically designed for autonomous driving and provides images, camera poses, and High-Definition (HD) maps. For their experiments, they focus on front-facing cameras (left-rear, center, and right-rear) and select six sequences from the dataset, using four for detailed evaluation and visualization.

Key Insights and Ideas

  • Map Priors as Weak Supervision: The core insight is that semantic map information, while not pixel-perfect, can serve as valuable "weak supervision" for NeRF training, especially for challenging tasks like view extrapolation. The key is to leverage this information intelligently, acknowledging its inherent uncertainties.
  • Uncertainty is Key: Explicitly modeling and managing uncertainty in the map priors is crucial. This allows the model to learn robustly from potentially noisy or incomplete map data and to handle the inherent ambiguity in extrapolated views.
  • Semantic Consistency is Paramount for AD: For AD simulations, maintaining semantic consistency in synthesized views is as important, if not more so, than photorealism. MapNeRF's focus on this aspect directly addresses a critical limitation of prior NeRF methods in this domain.
  • Gradual Exploration: The uncertainty tempering scheme is an elegant way to balance the guidance from map priors with the need for the NeRF to learn scene geometry and appearance from scratch, especially in novel, extrapolated views.

Key Results

  • Improved Extrapolation: MapNeRF demonstrates significant improvements in out-of-trajectory view extrapolation compared to a strong NeRF baseline (Nerfacto, referred to as "NeRF-base"). Quantitative results show an increase of 2.06 PSNR and improvements in SSIM and LPIPS metrics for extrapolation tasks.
  • Enhanced Semantic Consistency: Qualitative results, particularly in Figure 4 and Figure 5, show that MapNeRF produces more semantically consistent and accurate renderings of road features (like lanes) in deviated views. Figure 5, for instance, shows that synthesized views from MapNeRF allow for more accurate lane detection by a downstream model.
  • Comparable Interpolation: For in-trajectory view interpolation, MapNeRF performs comparably to, and slightly better than, the baseline, indicating that the added map priors do not degrade performance on standard tasks.
  • Ablation Study Validation: An ablation study (Table III) confirms the effectiveness of each proposed component (ground density supervision, multi-view consistency, and uncertainty tempering), showing consistent improvements as each is added.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that quantitative evaluation of view extrapolation is challenging due to the lack of perfect ground truth data for deviated views. They use a proxy method of holding out frames from lane-change sequences for evaluation.
  • The current method relies on pre-built HD maps, which may not always be available or perfectly accurate.
  • Future Work:
    • The authors aim to explore the integration of perception algorithms directly into self-driving cars using real-time neural radiance fields.
  • This could involve incorporating techniques like semantic segmentation, lane detection, and dynamic obstacle in-painting to further enhance the realism and utility of the simulated environments.
  • The uncertainty parameter is highlighted as a mechanism that could be extended to incorporate other types of priors beyond just geometric map data.

Summary updated

Abstract

Simulating camera sensors is a crucial task in autonomous driving. Although neural radiance fields are exceptional at synthesizing photorealistic views in driving simulations, they still fail to generate extrapolated views. This paper proposes to incorporate map priors into neural radiance fields to synthesize out-of-trajectory driving views with semantic road consistency. The key insight is that map information can be utilized as a prior to guiding the training of the radiance fields with uncertainty. Specifically, we utilize the coarse ground surface as uncertain information to supervise the density field and warp depth with uncertainty from unknown camera poses to ensure multi-view consistency. Experimental results demonstrate that our approach can produce semantic consistency in deviated views for vehicle camera simulation. The supplementary video can be viewed at https://youtu.be/jEQWr-Rfh3A.
BibTeX
@inproceedings{wu2023mapnerf,
  author = {Chenming Wu and Jiadai Sun and Zhelun Shen and Liangjun Zhang},
  title = {{MapNeRF: Incorporating Map Priors into Neural Radiance Fields for Driving View Simulation}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=7xRHtbZnfp},
}
Figure