psPRF:Pansharpening Planar Neural Radiance Field for Generalized 3D Reconstruction Satellite Imagery

Tongtong Zhang, Yuanxiang Li

TL;DR

psPRF is a novel framework that merges two traditionally separate satellite imaging tasks—pan-sharpening and 3D reconstruction—into a single, end-to-end process. By utilizing a multimodal planar radiance field, it can generate high-resolution 3D models and novel views from a single pair of low-resolution color and high-resolution panchromatic images, demonstrating impressive generalization across different geographical scenes.

What Problem Does This Paper Address?

Satellite imaging faces a fundamental hardware trade-off: you can have high spectral resolution (rich color but blurry) or high spatial resolution (sharp details but grayscale). To get the best of both worlds, researchers usually perform pan-sharpening as a preprocessing step to fuse these images.

Furthermore, while Neural Radiance Fields (NeRFs) have revolutionized 3D reconstruction, most satellite-specific variants (like SatNeRF) are "overfit" to a single scene. They require dozens of images of the same spot and hours of training to reconstruct just one small area. This paper tackles the lack of generalization (the ability to work on new, unseen locations) and the inefficiency of treating pan-sharpening and 3D reconstruction as disjointed problems.

Methods

The authors introduce the Pansharpening Planar Neural Radiance Field (psPRF). Its architecture is built on a few key innovations:

  1. Multimodal Encoder with SSConv: The model uses a U-Net-style encoder that takes both Low-Resolution RGB (LR-RGB) and High-Resolution Panchromatic (HR-PAN) images. It employs Spectral-to-Spatial Convolution (SSConv) to upsample the color features while aligning them with the sharp spatial details of the PAN image.
  2. Planar Radiance Field (PRF): Instead of the standard NeRF approach of sampling thousands of points along individual rays (which is computationally expensive), psPRF uses a Multiplane Image (MPI) representation. It predicts a stack of fronto-parallel planes, each containing color, PAN intensity, and volume density.
  3. Differentiable RPC Warping: Satellite cameras use Rational Polynomial Camera (RPC) models rather than simple pinhole models. The authors developed a differentiable way to warp these image planes into 3D "object space" using RPC coefficients, allowing the model to maintain geometric consistency across different viewpoints.
  4. Self-Supervised Reprojection Loss: To ensure the model learns actual 3D geometry rather than just "painting" a flat image, they use a reprojection loss. The model synthesizes a view, projects it into 3D space, and then reprojects it back to a different camera's perspective to see if it matches the source data.

Datasets

The study utilizes high-resolution imagery from the Maxar WorldView-3 satellite. Specifically, they used subsets from the DFC2019 dataset, focusing on two primary locations:

  • Jacksonville (JAX): Subsets JAX_033 and JAX_070.
  • Omaha (OMA): Subsets OMA_042 and OMA_181.

These datasets provided the ground truth for both high-resolution color images and Digital Elevation Models (DSMs) used to validate the 3D reconstruction accuracy.

Key Insights and Ideas

  • Pan-sharpening as a Byproduct: The authors realized that 3D geometry provides a powerful prior for image fusion. By forcing the network to output both RGB and PAN intensities within a 3D volume, the color information naturally "finds" its high-resolution spatial home.
  • Efficiency via Planes: By treating the scene as a stack of planes rather than a continuous volume of points, the model achieves a massive speedup in inference time, making it much more practical for large-scale Earth observation.
  • Generalization is Possible: Unlike vanilla NeRFs, psPRF doesn't need to be retrained from scratch for every new city. Once trained on a variety of scenes, it can perform "zero-shot" or "few-shot" reconstruction on new geographical areas.
  • The "Depth Loss" Paradox: Interestingly, the authors found that adding ground-truth depth supervision actually decreased the quality of the synthesized images. They attribute this to potential inaccuracies in the ground-truth DSMs or a mismatch in optimization speeds between geometry and color.

Key Results

  • Superior Sparse-View Performance: In tests with only 1 or 3 input views, psPRF significantly outperformed SOTA models like EO-NeRF and SatensoRF, which often fail completely when data is sparse.
  • Inference Speed: psPRF is roughly 30 times faster than EO-NeRF (0.10 seconds vs. 3.17 seconds per image triple), thanks to its planar rendering approach.
  • Altitude Accuracy: The model achieved a Mean Absolute Error (MAE) in altitude estimation of approximately 3.05 meters in an entirely unsupervised manner (using only image consistency as a guide).
  • Visual Quality: Quantitative metrics like PSNR and SSIM were consistently higher for psPRF across both single-scene and multi-scene generalization tests.

Limitations and Future Work

While psPRF is a major step forward, the authors note a few areas for improvement:

  • Unsynchronized Optimization: The model sometimes converges on "transmittance" (transparency) faster than it does on the MPI weights, which can lead to artifacts.
  • Unimodal Distribution: Future work aims to encourage the MPI decoder to produce a more "unimodal" distribution (meaning the model is more certain about exactly where the ground surface is), which should further sharpen the resulting 3D models.
  • Handling Changing Scenes: While it handles sparse views well, extremely dynamic scenes (moving cars, changing shadows over long periods) still pose a challenge for the planar assumption.

Summary updated

Abstract

Most current NeRF variants for satellites are designed for one specific scene and fall short of generalization to new geometry. Additionally, the RGB images require pan-sharpening as an independent preprocessing step. This paper introduces psPRF, a Planar Neural Radiance Field designed for paired low-resolution RGB (LR-RGB) and high-resolution panchromatic (HR-PAN) images from satellite sensors with Rational Polynomial Cameras (RPC). To capture the cross-modal prior from both of the LR-RGB and HR-PAN images, for the Unet-shaped architecture, we adapt the encoder with explicit spectral-to-spatial convolution (SSConv) to enhance the multimodal representation ability. To support the generalization ability of psRPF across scenes, we adopt projection loss to ensure strong geometry self-supervision. The proposed method is evaluated with the multi-scene WorldView-3 LR-RGB and HR-PAN pairs, and achieves state-of-the-art performance.
BibTeX
@misc{zhang2024psprf,
  author = {Tongtong Zhang and Yuanxiang Li},
  title = {{psPRF:Pansharpening Planar Neural Radiance Field for Generalized 3D Reconstruction Satellite Imagery}},
  year = {2024},
  eprint = {2406.15707},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.15707},
}
Figure