Multiplane Prior Guided Few-Shot Aerial Scene Rendering

Zihan Gao, Licheng Jiao, Lingling Li, Xu Liu, Fang Liu, Puhua Chen, Yuwei Guo

Abstract

Neural Radiance Fields (NeRF) have transformed 3D scene reconstruction but typically require dense multi-view inputs to avoid artifacts. In aerial scenarios, data acquisition is often constrained, leading to "few-shot" challenges where standard NeRFs produce blurry geometry and "floaters." This paper introduces MPNeRF, a hybrid framework that leverages a Multiplane Image (MPI) representation as a geometric prior to guide NeRF training. By distilling local structural knowledge from an MPI branch into a continuous NeRF volume, MPNeRF achieves high-fidelity aerial rendering from as few as three images, significantly outperforming existing state-of-the-art methods in data efficiency and visual quality.

Introduction

Reconstructing 3D environments from Unmanned Aerial Vehicle (UAV) imagery is critical for applications ranging from urban planning to disaster response. While Neural Radiance Fields (NeRF) offer a powerful way to represent these scenes, their performance degrades sharply when the number of input views is limited. In these "few-shot" scenarios, the optimization process often overfits to the sparse training views, resulting in "floaters"—erroneous density in empty space—and a lack of structural coherence.

Aerial scenes present a unique opportunity: they are often characterized by relatively planar landscapes viewed from a bird's-eye perspective. MPNeRF exploits this domain-specific geometry by combining the strengths of Multiplane Images (MPI) and traditional NeRF architectures.

Methodology

The Dual-Branch Architecture

MPNeRF utilizes a dual-branch system designed to balance local detail with global consistency:

  1. The MPI Branch: This branch represents the scene as a stack of semi-transparent 2D layers at varying depths. It employs a SwinV2 Transformer encoder (pre-trained via SimMIM) to extract rich features. The MPI representation is particularly adept at capturing high-frequency details and local structures in aerial landscapes due to their naturally layered nature.
  2. The NeRF Branch: This branch maintains a standard continuous volumetric representation. While NeRF excels at global 3D consistency, it struggles to define sharp boundaries when data is sparse.

Multiplane Prior Guidance

The core innovation of MPNeRF is the "Multiplane Prior." During the training phase, the MPI branch generates "pseudo-labels"—predicted color and density values—for viewpoints that are not present in the training set. These pseudo-labels serve as a supervisory signal for the NeRF branch. By training the NeRF to match the MPI’s predictions on these unseen views, the model effectively distills the MPI’s understanding of local geometry into the NeRF’s continuous volume, preventing the geometric collapse typically seen in sparse-view reconstructions.

Experimental Results

Dataset and Setup

The researchers evaluated MPNeRF using the LEVIR-NVS dataset, which consists of 16 diverse aerial scenes, including urban centers, colleges, and mountainous terrain. To simulate extreme data scarcity, the model was tested in 3-view and 5-view configurations.

Quantitative Performance

MPNeRF established a new state-of-the-art for few-shot aerial rendering:

  • Accuracy: In 3-view settings, MPNeRF achieved a PSNR of 21.72, a massive improvement over the 15.13 achieved by vanilla NeRF. It also showed significant gains in structural metrics like SSIM and LPIPS.
  • Efficiency: The framework demonstrated remarkable data efficiency, achieving rendering quality comparable to standard NeRF while requiring 63.5% fewer training images.

Qualitative Advantages

Visual comparisons highlight the model's ability to eliminate the "ghosting" and blurriness prevalent in other few-shot methods like DietNeRF or InfoNeRF. MPNeRF maintains sharp edges on man-made structures (e.g., buildings and roads) and preserves the intricate textures of natural landscapes.

Discussion and Limitations

While MPNeRF represents a significant leap forward, it faces certain challenges:

  • Computational Overhead: The memory requirements for the MPI branch scale quadratically with image resolution, which may limit its application for massive, city-scale reconstructions on standard hardware.
  • Error Propagation: The NeRF branch's reliance on the MPI branch means that errors in the MPI's initial predictions—particularly in highly complex, non-planar regions—can propagate into the final 3D model.

Conclusion

MPNeRF demonstrates that incorporating domain-specific geometric priors can overcome the data-hungry nature of neural rendering. By bridging the gap between discrete multiplane representations and continuous volumetric fields, the framework enables high-quality 3D aerial reconstruction from minimal data, paving the way for more efficient UAV-based mapping and visualization.

Summary updated

Abstract

Neural Radiance Fields (NeRF) have been successfully applied in various aerial scenes, yet they face challenges with sparse views due to limited supervision. The acquisition of dense aerial views is often prohibitive, as unmanned aerial vehicles (UAVs) may encounter constraints in perspective range and energy constraints. In this work, we introduce Multiplane Prior guided NeRF (MPNeRF), a novel approach tailored for few-shot aerial scene rendering-marking a pioneering effort in this domain. Our key insight is that the intrinsic geometric regularities specific to aerial imagery could be leveraged to enhance NeRF in sparse aerial scenes. By investigating NeRF's and Multiplane Image (MPI)'s behavior, we propose to guide the training process of NeRF with a Multiplane Prior. The proposed Multiplane Prior draws upon MPI's benefits and incorporates advanced image comprehension through a Swin V2 Transformer, pre-trained via SimMIM. Our extensive experiments demonstrate that MPN-eRF outperforms existing state-of-the-art methods applied in non-aerial contexts, by tripling the performance in SSIM and LPIPS even with three views available. We hope our work offers insights into the development of NeRF-based applications in aerial scenes with limited data.
BibTeX
@inproceedings{gao2024multiplane,
  author = {Zihan Gao and Licheng Jiao and Lingling Li and Xu Liu and Fang Liu and Puhua Chen and Yuwei Guo},
  title = {{Multiplane Prior Guided Few-Shot Aerial Scene Rendering}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
}
Figure