Abstract
Standard deep stereo networks often struggle when transitioning from synthetic training environments to real-world scenarios due to over-specialization on image appearance. This paper introduces Matching-Space Networks (MS-Nets), which utilize domain-invariant matching functions instead of learned RGB feature extraction. By forcing the model to learn geometry rather than color-specific features, MS-Nets achieve significantly improved performance and robustness on unseen datasets.
Introduction
Stereo matching, the process of estimating depth by finding correspondences between image pairs, has seen significant advancements through end-to-end deep learning. However, a persistent "generalization gap" remains: models trained on synthetic data (e.g., SceneFlow) frequently fail when applied to real-world datasets like KITTI or Middlebury. This failure is largely attributed to over-specialization, where networks learn to match pixels based on domain-specific appearance properties like lighting and texture.
Methodology: From Color Space to Matching Space
The core contribution of this work is the transition from raw Color Space to a structured Matching Space.
Fixed Feature Extraction
Instead of relying on a CNN to learn features from RGB values, MS-Nets employ four traditional, domain-invariant matching functions:
- Normalized Cross Correlation (NCC)
- Zero-mean Sum of Absolute Differences (ZSAD)
- Census Transform
- Sobel Operator
Matching Space (MS) Representation
For each pixel and disparity level, the network calculates matching costs and confidence measures (likelihoods) for these four functions. This results in an 8-channel feature vector per pixel-disparity pair, forming a 4D "Matching Volume."
Hybrid Architecture
This Matching Volume is processed by state-of-the-art 3D CNN backends, such as variants of GCNet and PSMNet. By abstracting the input into mathematically robust descriptors like the Census transform, the network is shielded from domain-specific appearance shifts and forced to learn the underlying structural relationships of the matching problem.
Experimental Evaluation
The authors evaluated MS-Nets across several diverse datasets including SceneFlow for initial training, and KITTI, Middlebury 2014, and ETH3D for cross-domain testing. They also assessed performance on "scarce data" subsets to evaluate stability under limited supervision.
Key Results and Discussion
MS-Nets demonstrate superior generalization capabilities. When trained on synthetic data and tested on the Middlebury dataset, MS-GCNet achieved an error rate (bad2-noc) of 18.52%, a substantial improvement over the 30.42% error rate of the standard RGB-based GCNet.
Qualitative Improvements
Qualitatively, MS-Nets excel at recovering fine details and handling large planar surfaces. Standard models often produce noisy or "bumpy" disparity maps in unseen domains, whereas MS-Nets maintain structural integrity.
Robustness to Data Scarcity
A significant finding is the stability of MS-Nets under limited supervision. While standard networks yield unpredictable results when trained on small datasets, MS-Nets remain consistent because their input is already a highly structured representation of the geometry.
Conclusion and Future Work
While MS-Nets offer exceptional generalization, they face a slight "specialization tax" when tested on the same domain used for training, as they cannot exploit domain-specific RGB cues. Future research may explore learning a set of domain-invariant matching functions or making the selection process differentiable. Despite a small computational overhead for pre-processing (~289ms), MS-Nets provide a robust solution for real-world stereo vision applications.
Summary updated