Remote Sensing Novel View Synthesis With Implicit Multiplane Representations

Yongchang Wu, Zhengxia Zou, Zhenwei Shi

TL;DR

This paper introduces ImMPI, a novel method for generating new views of remote sensing scenes using implicit multiplane representations. By combining implicit neural networks with explicit multiplane images and a novel cross-scene initialization technique, ImMPI achieves faster rendering and more accurate novel view synthesis, especially in challenging sparse-view scenarios.

What Problem Does This Paper Address?

Generating realistic new views of remote sensing (RS) scenes from existing images is crucial for applications like visualization and analysis. However, RS imagery presents unique challenges: high scene complexity, limited variations in camera viewpoints (view sparsity), and significant depth variations. Existing methods, often based on volumetric rendering (like NeRF), struggle with these challenges, leading to slow rendering, difficulty in handling large-scale scenes, and overfitting when training data is sparse. This paper aims to bridge this gap by developing a method that is both efficient and robust for RS novel view synthesis.

Methods

The core of the proposed method, ImMPI, lies in its hybrid representation and a two-stage optimization process:

  • Implicit Multiplane Images (ImMPI): This combines the strengths of implicit neural representations (encoding scene geometry efficiently in network weights) with explicit Multiplane Images (MPIs). MPIs are well-suited for RS due to the typically perpendicular camera view, representing the scene as a series of RGBA planes. This hybrid approach aims for faster rendering than pure volumetric methods.
  • Differentiable Multiplane Renderer: The system uses a differentiable renderer to reconstruct the 3D scene from multi-view inputs. This allows for self-supervised optimization by comparing rendered views with ground truth.
  • Cross-Scene Neural Network Initialization: To address the issue of sparse views and accelerate per-scene optimization, the authors introduce a "Prior extractor" network. This network is trained on a large dataset of diverse RS scenes to learn general 3D scene priors. These priors are then used to initialize the ImMPI for a new scene, providing a strong starting point and regularizing the optimization process.
  • Per-Scene Optimization: After initialization, the ImMPI is further refined using images from the specific scene. This stage focuses on optimizing the MPI generator network's weights rather than individual MPI pixels, which helps prevent overfitting and ensures smoother results.

Datasets

  • WHU MVS/Stereo Dataset: Used for training the cross-scene initialization network. This dataset contains multi-view images suitable for depth estimation tasks.
  • LEVIR-NVS Dataset: A new dataset introduced by the authors for remote sensing novel view synthesis. It comprises 16 real-world 3D scenes from Google Earth, each with 21 multi-view images. This dataset is used for per-scene optimization and evaluation.

Key Insights and Ideas

  • MPIs are a natural fit for Remote Sensing: The paper highlights that the typically overhead view of RS imagery makes MPIs a more intuitive and efficient representation compared to volumetric approaches.
  • Cross-scene initialization is crucial for sparse views: Learning general scene priors from diverse datasets significantly improves the stability and speed of per-scene optimization, especially when only a few input views are available. This acts as a powerful regularizer.
  • Hybrid implicit-explicit representation balances efficiency and expressiveness: Combining the compact nature of implicit networks with the directness of MPIs allows for both efficient representation and fast rendering.
  • Depth estimation as a byproduct: The ImMPI framework naturally allows for the generation of depth maps alongside novel views, adding further utility to the method.

Key Results

  • Improved Rendering Fidelity: Qualitative comparisons show ImMPI producing sharper, more detailed, and artifact-free novel views compared to NeRF and NeRF++ on the LEVIR-NVS dataset, especially for test views.
  • Faster Optimization: ImMPI achieves per-scene optimization in approximately 30 minutes, significantly faster than NeRF and NeRF++ which take over an hour.
  • Extremely Fast Rendering: Novel views can be rendered in under 1 second per frame, which is at least 20 times faster than other methods.
  • Robustness to Sparse Views: The method demonstrates satisfactory results even with as few as 3 training views, highlighting its effectiveness in challenging sparse-view scenarios.
  • Quantitative Improvements: ImMPI consistently outperforms NeRF and NeRF++ in PSNR, SSIM, and LPIPS metrics on test views. For example, the mean PSNR on test views is 25.95, compared to 12.68 for NeRF and 17.96 for NeRF++.

Limitations and Future Work

  • Pre-training Time: The cross-scene initialization requires a significant pre-training phase (21 hours), although this is amortized over many scenes.
  • Depth Estimation Artifacts: While depth estimation is a byproduct, the paper notes that sharp depth changes can still lead to some artifacts in the generated depth maps.
  • Future Work: The authors suggest that their method can be extended to handle more complex camera movements and potentially larger-scale scenes. Further exploration of the depth hypothesis number could also yield improvements. The introduction of the LEVIR-NVS dataset also opens avenues for future research in this domain.

Summary updated

Abstract

Novel view synthesis of remote sensing (RS) scenes is of great significance for scene visualization, human–computer interaction, and various downstream applications. Despite the recent advances in computer graphics and photogrammetry technology, generating novel views is still challenging particularly for RS images due to its high complexity, view sparsity, and limited view-perspective variations. In this article, we propose a novel RS view synthesis method by leveraging the recent advances in implicit neural representations. Considering the overhead and far depth imaging of RS images, we represent the 3-D space by combining implicit multiplane images (ImMPI) representation and deep neural networks. The 3-D scene is reconstructed under a self-supervised optimization paradigm through a differentiable multiplane renderer with multiview input constraints. Images from any novel views thus can be freely rendered on the basis of the reconstructed model. As a by-product, the depth maps corresponding to the given viewpoint can be generated along with the rendering output. We refer to our method as ImMPIs. To further improve the view synthesis under sparse-view inputs, we explore the learning-based initialization of RS 3-D scenes and proposed a neural-network-based prior extractor to accelerate the optimization process. In addition, we propose a new dataset for RS novel view synthesis with multiview real-world Google Earth images. Extensive experiments demonstrate the superiority of the ImMPI over previous state-of-the-art methods in terms of reconstruction accuracy, visual fidelity, and time efficiency. Ablation experiments also suggest the effectiveness of our methodology design.
BibTeX
@article{wu2022remote,
  author = {Yongchang Wu and Zhengxia Zou and Zhenwei Shi},
  title = {{Remote Sensing Novel View Synthesis With Implicit Multiplane Representations}},
  journal = {IEEE Trans. Geosci. Remote. Sens.},
  year = {2022},
  url = {https://openreview.net/forum?id=Vwsp-IWBL0-},
}
Figure