DSM Accuracy Evaluation for the ISPRS Commission I Image Matching Benchmark

G. Kuschk, Pablo d’Angelo, Rongjun Qin, Daniela Poli, Peter Reinartz, Daniel Cremers2014 · DOI

Abstract: To improve the quality of algorithms for automatic generation of Digital Surface Models (DSM) from optical stereo data, the ISPRS Working Group 4 of Commission I provides a benchmark dataset for measuring and comparing the accuracy of dense stereo algorithms. The data consists of optical spaceborne stereo images together with ground truth data produced by aerial laser scanning. In this paper, we present improvements to this benchmark. First, to lower the barrier for researchers, we provide rectified input images that enable stereo algorithms to work without implementing complex satellite camera models. Second, we improved evaluation robustness against orientation errors using Iterative Closest Point (ICP) alignment. Finally, we implemented an adaptive triangulation method for point-to-surface evaluation, allowing for more accurate measurement of non-uniformly sampled DSMs and complex 3D models. We demonstrate these methods by comparing three matching algorithms across different land cover classes.

Introduction: The Challenge of Satellite Stereo Matching

Generating accurate 3D models of the Earth's surface from satellite stereo pairs is a cornerstone of modern remote sensing. However, two major hurdles prevent many researchers from entering this field:

  1. Complexity of Satellite Geometry: Unlike standard cameras, satellites use "push-broom" sensors with complex Rational Polynomial Coefficient (RPC) models. Integrating these into standard stereo-matching frameworks is a significant technical burden.
  2. Evaluation Bias: Comparing a generated DSM to ground truth (like LiDAR) is sensitive to sensor alignment. If the satellite's orientation is slightly off, the resulting error reflects poor sensor alignment rather than a failure of the matching algorithm itself.

Methodology

The authors propose a modernized approach to the benchmark to address these issues and encourage participation from the broader computer vision community.

The proposed workflow for image rectification and DSM evaluation against LiDAR ground truth.

Epipolar Rectification

A "two-step" rectification process transforms raw satellite images into a standard geometry where matching correspondences lie on the same horizontal row. This allows researchers to use "off-the-shelf" stereo algorithms without needing to understand satellite-specific camera models.

Robust Alignment (ICP)

To ensure they are measuring matching accuracy rather than orientation errors, the authors use the Iterative Closest Point (ICP) algorithm. This aligns the generated 3D point cloud to the LiDAR ground truth in 3D space before calculating errors, effectively decoupling reconstruction quality from absolute georeferencing accuracy.

Adaptive Triangulation

Instead of comparing height values at fixed grid points, the authors triangulate the generated DSM using an adaptive method that minimizes surface curvature. They then calculate the Euclidean distance from each LiDAR point to the nearest triangle on the DSM surface. This "point-to-surface" approach is superior for handling non-uniformly sampled data and sharp architectural features.

Datasets

The benchmark utilizes high-resolution optical satellite data covering three characteristic regions near Barcelona, Spain (Terrassa, Vacarisses, and La Mola):

  • Cartosat-1: 2.5m Ground Sample Distance (GSD).
  • Worldview-1: 0.5m to 0.76m GSD.
  • Pleiades: A triplet of images with varying off-nadir angles.
  • Ground Truth: High-accuracy LiDAR point clouds (0.3 points per square meter) and 50cm color orthoimages.

Key Results and Insights

The authors tested baseline algorithms including Semi-Global Matching (SGM) and a Total Variation (TV) minimization approach.

Visual comparison of the DSM generated via SGM, the LiDAR ground truth, and the resulting error map.
  • Resolution and Accuracy: Higher-resolution Worldview-1 data yielded significantly better results than Cartosat-1. On Worldview-1, the TV-based approach achieved a Mean Absolute Error (MAE) of 1.87m, while SGM achieved 2.22m.
  • Land Cover Impact: Algorithm performance varies by terrain. In the urban "Terrassa" area, errors were higher in residential and city center zones compared to open fields, highlighting the difficulty of matching in areas with high occlusion and complex vertical structures.
  • Surface-Based Evaluation Benefits: Moving to 3D point-to-surface distances allows for a fairer evaluation of 3D models, especially when those models are used for building extraction or data reduction.

Conclusion

By providing rectified images and a robust evaluation pipeline, the authors have lowered the barrier to entry for satellite-based 3D reconstruction. Future work will involve evaluating state-of-the-art cost functions and more sophisticated energy minimization methods to further push the boundaries of DSM accuracy.

Summary updated

Abstract

To improve the quality of algorithms for automatic generation of Digital Surface Models (DSM) from optical stereo data in the remote sensing community, the Working Group 4 of Commission I: Geometric and Radiometric Modeling of Optical Airborne and Spaceborne Sensors provides on its website http://www2.isprs.org/commissions/comm1/wg4/benchmark-test.html a benchmark dataset for measuring and comparing the accuracy of dense stereo algorithms. The data provided consists of several optical spaceborne stereo images together with ground truth data produced by aerial laser scanning. In this paper we present our latest work on this benchmark, based upon previous work. As a first point, we noticed that providing the abovementioned test data as geo-referenced satellite images together with their corresponding RPC camera model seems too high a burden for being used widely by other researchers, as a considerable effort still has to be made to integrate the test datas camera model into the researchers local stereo reconstruction framework. To bypass this problem, we now also provide additional rectified input images, which enable stereo algorithms to work out of the box without the need for implementing special camera models. Care was taken to minimize the errors resulting from the rectification transformation and the involved image resampling. We further improved the robustness of the evaluation method against errors in the orientation of the satellite images (with respect to the LiDAR ground truth). To this end we implemented a point cloud alignment of the DSM and the LiDAR reference points using an Iterative Closest Point (ICP) algorithm and an estimation of the best fitting transformation. This way, we concentrate on the errors from the stereo reconstruction and make sure that the result is not biased by errors in the absolute orientation of the satellite images. The evaluation of the stereo algorithms is done by triangulating the resulting (filled) DSMs and computing for each LiDAR point the nearest Euclidean distance to the DSM surface. We implemented an adaptive triangulation method minimizing the second order derivative of the surface in a local neighborhood, which captures the real surface more accurate than a fixed triangulation. As a further advantage, using our point-to-surface evaluation, we are also able to evaluate non-uniformly sampled DSMs or triangulated 3D models in general. The latter is for example needed when evaluating building extraction and data reduction algorithms. As practical example we compare results from three different matching methods applied to the data available within the benchmark data sets. These results are analyzed using the above mentioned methodology and show advantages and disadvantages of the different methods, also depending on the land cover classes.
BibTeX
@inproceedings{kuschk2014dsm,
  author = {G. Kuschk and Pablo d’Angelo and Rongjun Qin and Daniela Poli and Peter Reinartz and Daniel Cremers},
  title = {{DSM Accuracy Evaluation for the ISPRS Commission I Image Matching Benchmark}},
  booktitle = {˜The},
  year = {2014},
  doi = {10.5194/isprsarchives-xl-1-195-2014},
}
Figure