Abstract
Neural Radiance Fields (NeRF) have transformed 3D scene reconstruction but typically require dense multi-view inputs to avoid artifacts. In aerial scenarios, data acquisition is often constrained, leading to "few-shot" challenges where standard NeRFs produce blurry geometry and "floaters." This paper introduces MPNeRF, a hybrid framework that leverages a Multiplane Image (MPI) representation as a geometric prior to guide NeRF training. By distilling local structural knowledge from an MPI branch into a continuous NeRF volume, MPNeRF achieves high-fidelity aerial rendering from as few as three images, significantly outperforming existing state-of-the-art methods in data efficiency and visual quality.
Introduction
Reconstructing 3D environments from Unmanned Aerial Vehicle (UAV) imagery is critical for applications ranging from urban planning to disaster response. While Neural Radiance Fields (NeRF) offer a powerful way to represent these scenes, their performance degrades sharply when the number of input views is limited. In these "few-shot" scenarios, the optimization process often overfits to the sparse training views, resulting in "floaters"—erroneous density in empty space—and a lack of structural coherence.
Aerial scenes present a unique opportunity: they are often characterized by relatively planar landscapes viewed from a bird's-eye perspective. MPNeRF exploits this domain-specific geometry by combining the strengths of Multiplane Images (MPI) and traditional NeRF architectures.
Methodology
The Dual-Branch Architecture
MPNeRF utilizes a dual-branch system designed to balance local detail with global consistency:
- The MPI Branch: This branch represents the scene as a stack of semi-transparent 2D layers at varying depths. It employs a SwinV2 Transformer encoder (pre-trained via SimMIM) to extract rich features. The MPI representation is particularly adept at capturing high-frequency details and local structures in aerial landscapes due to their naturally layered nature.
- The NeRF Branch: This branch maintains a standard continuous volumetric representation. While NeRF excels at global 3D consistency, it struggles to define sharp boundaries when data is sparse.
Multiplane Prior Guidance
The core innovation of MPNeRF is the "Multiplane Prior." During the training phase, the MPI branch generates "pseudo-labels"—predicted color and density values—for viewpoints that are not present in the training set. These pseudo-labels serve as a supervisory signal for the NeRF branch. By training the NeRF to match the MPI’s predictions on these unseen views, the model effectively distills the MPI’s understanding of local geometry into the NeRF’s continuous volume, preventing the geometric collapse typically seen in sparse-view reconstructions.
Experimental Results
Dataset and Setup
The researchers evaluated MPNeRF using the LEVIR-NVS dataset, which consists of 16 diverse aerial scenes, including urban centers, colleges, and mountainous terrain. To simulate extreme data scarcity, the model was tested in 3-view and 5-view configurations.
Quantitative Performance
MPNeRF established a new state-of-the-art for few-shot aerial rendering:
- Accuracy: In 3-view settings, MPNeRF achieved a PSNR of 21.72, a massive improvement over the 15.13 achieved by vanilla NeRF. It also showed significant gains in structural metrics like SSIM and LPIPS.
- Efficiency: The framework demonstrated remarkable data efficiency, achieving rendering quality comparable to standard NeRF while requiring 63.5% fewer training images.
Qualitative Advantages
Visual comparisons highlight the model's ability to eliminate the "ghosting" and blurriness prevalent in other few-shot methods like DietNeRF or InfoNeRF. MPNeRF maintains sharp edges on man-made structures (e.g., buildings and roads) and preserves the intricate textures of natural landscapes.
Discussion and Limitations
While MPNeRF represents a significant leap forward, it faces certain challenges:
- Computational Overhead: The memory requirements for the MPI branch scale quadratically with image resolution, which may limit its application for massive, city-scale reconstructions on standard hardware.
- Error Propagation: The NeRF branch's reliance on the MPI branch means that errors in the MPI's initial predictions—particularly in highly complex, non-planar regions—can propagate into the final 3D model.
Conclusion
MPNeRF demonstrates that incorporating domain-specific geometric priors can overcome the data-hungry nature of neural rendering. By bridging the gap between discrete multiplane representations and continuous volumetric fields, the framework enables high-quality 3D aerial reconstruction from minimal data, paving the way for more efficient UAV-based mapping and visualization.
Summary updated