TL;DR
This paper experimentally compares various multi-view stereo (MVS) methods for generating Digital Surface Models (DSMs) from satellite imagery. It finds that while deep learning (DL) methods show promise, adapted classic pair-wise stereo pipelines, especially when enhanced with DL for matching, often outperform true MVS DL methods on satellite data due to the unique challenges of this domain.
What Problem Does This Paper Address?
Generating accurate 3D elevation maps (Digital Surface Models or DSMs) from satellite imagery is crucial for various applications. While multi-view stereo (MVS) techniques have advanced significantly in computer vision for close-range scenes, their adaptation to satellite imagery presents unique challenges. These include the extreme difference between depth range and camera distance, and significant variations in illumination and appearance due to long acquisition times between images. Traditional satellite stereo pipelines often rely on pair-wise classic algorithms, and this paper investigates whether more modern, deep learning-based MVS methods can offer improvements, either as standalone solutions or as components within existing pipelines.
Methods
The authors evaluate several MVS methods, categorizing them into pair-wise and true multi-view approaches, and classic versus deep learning (DL) based techniques.
- S2P (Satellite Stereo Pipeline): A baseline modular pipeline for satellite image processing, using classic methods like MGM for disparity computation.
- S2P-GANet: This method integrates the GANet deep learning model into the S2P pipeline as a replacement for the classic stereo matching step. GANet uses a novel Semi-global Guided Aggregation (SGA) and Local Guided Aggregation (LGA) for efficient cost volume processing. Adaptations were made to handle GANet's requirement for negative disparities and GPU memory limitations through tiling.
- COLMAP: A well-established Structure-from-Motion (SfM) and MVS pipeline, adapted for satellite imagery by approximating RPC cameras with perspective models and reparameterizing depth.
- CasMVSNet: A true multi-view stereo method based on deep learning that uses a cascade of cost volumes to handle high-resolution images efficiently. It was adapted for satellite imagery by borrowing projection approximations from COLMAP's adaptation.
A key aspect of the evaluation framework is the DSM aggregation criteria for pair-wise methods, which includes both a priori filtering based on image metadata and a posteriori filtering based on the completeness of the computed DSM.
Datasets
The study uses three datasets:
- MVS3D: A set of 47 satellite images with airborne Lidar data as ground truth.
- US3D: Consists of WorldView-3 images from Jacksonville (JAX) and Omaha (OMA), also with airborne Lidar ground truth.
- For each dataset, five subregions were selected, with image sets acquired both in a short time interval (near-in-time, NIT) and over longer periods (far-in-time, FIT).
Key Insights and Ideas
- DL for Matching in Classic Pipelines is Promising: Integrating a DL stereo matching method like GANet into a classic pair-wise pipeline (S2P-GANet) shows significant improvements over the purely classic S2P pipeline, even without specific fine-tuning on satellite data. This suggests that DL can effectively enhance existing frameworks.
- True MVS DL Methods Struggle with Satellite Data: Despite their success in close-range computer vision, true MVS DL methods (like CasMVSNet) did not consistently outperform adapted pair-wise methods on satellite datasets. This is likely due to the inherent challenges of satellite imagery and the memory constraints of DL models for large-scale reconstructions.
- Generalization Power of GANet: The GANet model demonstrated good generalization capabilities, performing well on satellite data even when trained on general computer vision datasets.
- Importance of Pair Selection: For pair-wise MVS, effective aggregation of multiple stereo pairs is crucial, but careful selection is needed to avoid degrading results with poor matches, especially in the presence of significant seasonal changes. The a posteriori filtering criterion proved particularly effective.
Key Results
- S2P-GANet Outperforms S2P: On stereo pairs, S2P-GANet generally achieved comparable or better completeness and RMSE accuracy compared to the classic S2P pipeline. For instance, on the JAX_NIT dataset, S2P-GANet achieved an RMSE of 3.22 compared to S2P's 4.25.
- Pair-wise Methods Dominate: Aggregated pair-wise methods (S2P and S2P-GANet) generally outperformed true multi-view methods (COLMAP and CasMVSNet) across most datasets and metrics.
- COLMAP Adaptation: The adapted COLMAP showed reasonable performance but was generally surpassed by the pair-wise S2P variants.
- CasMVSNet Performance: The adapted CasMVSNet achieved metrics close to pair-wise MVS, indicating the potential of true MVS DL methods with proper adaptation, though not consistently outperforming the best pair-wise approaches.
Limitations and Future Work
- Dataset Scarcity: The paper highlights that the adaptation and development of methods for satellite imagery are hindered by the scarcity of large-scale aerial and satellite training datasets.
- Fine-tuning Benefits: Fine-tuning GANet on satellite-specific datasets (WHU and RVL) showed slight improvements in completeness and accuracy, suggesting that further domain-specific training could yield even better results.
- Real-world Application: The current evaluation relies on ground truth for registration. For real-world applications where ground truth is unavailable, a surrogate DSM would be needed for registration, as suggested by prior work.
- Future Directions: The authors suggest that the success of GANet as a matching component points to its potential as an attractive option for existing satellite pipelines. Further research into adapting and developing new DL-based MVS solutions tailored for satellite imagery is encouraged.
Summary updated