The HCI Stereo Metrics: Geometry-Aware Performance Analysis of Stereo Algorithms

Katrin Honauer, Lena Maier-Hein, Daniel Kondermann

Abstract

Traditional stereo vision metrics like Root Mean Squared Error (RMS) or "BadPix" provide a single average error value, often masking an algorithm's specific geometric strengths or weaknesses. This paper introduces the HCI Stereo Metrics, a suite of nine geometry-aware measures categorized into depth discontinuities, planar surfaces, and fine structures. By evaluating algorithms on semantically meaningful pixel subsets, the authors reveal that the "best" algorithm is highly context-dependent, enabling more informed selection for specialized applications like robotics or autonomous driving.

Introduction: The Limitations of "Blunt" Metrics

For over a decade, the stereo vision community has relied on the Middlebury benchmark and metrics like RMS. While easy to compute, these are "blunt instruments" that average errors across an entire image. They treat a flat wall the same as a complex tree branch or a sharp object edge. In real-world tasks, a small error on a flat surface might be negligible, but "fattening" an edge could cause a self-driving car to miscalculate a gap. Standard metrics fail to distinguish these nuances, often leading to rankings that hide an algorithm's true strengths or fatal flaws.

The HCI Stereo Metrics

The authors introduce nine semantically intuitive measures, automatically extracted from ground truth data based on three geometric cues:

  1. Depth Discontinuities: Measures include Foreground Fattening (edges bleeding into the background), Foreground Thinning (objects shrinking), and Fuzziness (noisy edges).
  2. Planar Surfaces: Using RANSAC to identify planes, the metrics assess Bumpiness (local noise), Offset (distance from the true plane), and Local Misorientation (angular error).
  3. Fine Structures: For thin objects, the metrics track Porosity (missing parts), Fragmentation (breaking objects apart), and Detail Fattening (thin structures becoming blobs).
Visualizing geometric errors: (a) fattening, (b) thinning, and (c) fuzziness at depth discontinuities.

Datasets and Evaluation

The study primarily utilizes the Middlebury Stereo Dataset (v.3), evaluating 13 top-performing algorithms. Comparisons are also made against results from KITTI, SINTEL, and HeiSt. By focusing on "meaningful" pixels—such as those on a thin wire—the metrics become significantly more sensitive than global averages.

Key Results: Ranking Flips and Independence

The most striking finding is that the "best" algorithm is context-dependent. An algorithm ranking #1 in overall RMS might be the worst at preserving sharp edges. The authors found the coefficient of determination ($r^2$) between RMS and metrics like Foreground Fattening to be as low as 0.04, confirming that traditional benchmarks are largely blind to edge-handling quality.

Algorithm performance varies significantly across different geometric features, leading to ranking flips compared to standard RMS metrics.

For instance, while one algorithm might excel at planar surfaces (crucial for surgical reconstruction), another might be superior for fine structures (essential for handling sutures). This allows for application-specific tuning, identifying the optimal tool for the task rather than relying on a generic leaderboard.

Limitations and Future Work

Current limitations include the lack of homogeneous normalization across different metric types and the need for manual parameter tuning during pixel subset extraction. Future work aims to address radiometric challenges like specular highlights and extend the framework to optical flow and multi-view stereo evaluation.

Summary updated

Abstract

Performance characterization of stereo methods is mandatory to decide which algorithm is useful for which application. Prevalent benchmarks mainly use the root mean squared error (RMS) with respect to ground truth disparity maps to quantify algorithm performance. We show that the RMS is of limited expressiveness for algorithm selection and introduce the HCI Stereo Metrics. These metrics assess stereo results by harnessing three semantic cues: depth discontinuities, planar surfaces, and fine geometric structures. For each cue, we extract the relevant set of pixels from existing ground truth. We then apply our evaluation functions to quantify characteristics such as edge fattening and surface smoothness. We demonstrate that our approach supports practitioners in selecting the most suitable algorithm for their application. Using the new Middlebury dataset, we show that rankings based on our metrics reveal specific algorithm strengths and weaknesses which are not quantified by existing metrics. We finally show how stacked bar charts and radar charts visually support multidimensional performance evaluation. An interactive stereo benchmark based on the proposed metrics and visualizations is available at: http://hci.iwr.uni-heidelberg.de/stereometrics.
BibTeX
@inproceedings{honauer2015hci,
  author = {Katrin Honauer and Lena Maier-Hein and Daniel Kondermann},
  title = {{The HCI Stereo Metrics: Geometry-Aware Performance Analysis of Stereo Algorithms}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2015},
  doi = {10.1109/iccv.2015.245},
}
Figure