Abstract
Traditional stereo vision metrics like Root Mean Squared Error (RMS) or "BadPix" provide a single average error value, often masking an algorithm's specific geometric strengths or weaknesses. This paper introduces the HCI Stereo Metrics, a suite of nine geometry-aware measures categorized into depth discontinuities, planar surfaces, and fine structures. By evaluating algorithms on semantically meaningful pixel subsets, the authors reveal that the "best" algorithm is highly context-dependent, enabling more informed selection for specialized applications like robotics or autonomous driving.
Introduction: The Limitations of "Blunt" Metrics
For over a decade, the stereo vision community has relied on the Middlebury benchmark and metrics like RMS. While easy to compute, these are "blunt instruments" that average errors across an entire image. They treat a flat wall the same as a complex tree branch or a sharp object edge. In real-world tasks, a small error on a flat surface might be negligible, but "fattening" an edge could cause a self-driving car to miscalculate a gap. Standard metrics fail to distinguish these nuances, often leading to rankings that hide an algorithm's true strengths or fatal flaws.
The HCI Stereo Metrics
The authors introduce nine semantically intuitive measures, automatically extracted from ground truth data based on three geometric cues:
- Depth Discontinuities: Measures include Foreground Fattening (edges bleeding into the background), Foreground Thinning (objects shrinking), and Fuzziness (noisy edges).
- Planar Surfaces: Using RANSAC to identify planes, the metrics assess Bumpiness (local noise), Offset (distance from the true plane), and Local Misorientation (angular error).
- Fine Structures: For thin objects, the metrics track Porosity (missing parts), Fragmentation (breaking objects apart), and Detail Fattening (thin structures becoming blobs).
Datasets and Evaluation
The study primarily utilizes the Middlebury Stereo Dataset (v.3), evaluating 13 top-performing algorithms. Comparisons are also made against results from KITTI, SINTEL, and HeiSt. By focusing on "meaningful" pixels—such as those on a thin wire—the metrics become significantly more sensitive than global averages.
Key Results: Ranking Flips and Independence
The most striking finding is that the "best" algorithm is context-dependent. An algorithm ranking #1 in overall RMS might be the worst at preserving sharp edges. The authors found the coefficient of determination ($r^2$) between RMS and metrics like Foreground Fattening to be as low as 0.04, confirming that traditional benchmarks are largely blind to edge-handling quality.
For instance, while one algorithm might excel at planar surfaces (crucial for surgical reconstruction), another might be superior for fine structures (essential for handling sutures). This allows for application-specific tuning, identifying the optimal tool for the task rather than relying on a generic leaderboard.
Limitations and Future Work
Current limitations include the lack of homogeneous normalization across different metric types and the need for manual parameter tuning during pixel subset extraction. Future work aims to address radiometric challenges like specular highlights and extend the framework to optical flow and multi-view stereo evaluation.
Summary updated