Matching-space Stereo Networks for Cross-domain Generalization

Changjiang Cai, Matteo Poggi, Stefano Mattoccia, Philippos Mordohai

Abstract

Standard deep stereo networks often struggle when transitioning from synthetic training environments to real-world scenarios due to over-specialization on image appearance. This paper introduces Matching-Space Networks (MS-Nets), which utilize domain-invariant matching functions instead of learned RGB feature extraction. By forcing the model to learn geometry rather than color-specific features, MS-Nets achieve significantly improved performance and robustness on unseen datasets.

Introduction

Stereo matching, the process of estimating depth by finding correspondences between image pairs, has seen significant advancements through end-to-end deep learning. However, a persistent "generalization gap" remains: models trained on synthetic data (e.g., SceneFlow) frequently fail when applied to real-world datasets like KITTI or Middlebury. This failure is largely attributed to over-specialization, where networks learn to match pixels based on domain-specific appearance properties like lighting and texture.

Methodology: From Color Space to Matching Space

The core contribution of this work is the transition from raw Color Space to a structured Matching Space.

Fixed Feature Extraction

Instead of relying on a CNN to learn features from RGB values, MS-Nets employ four traditional, domain-invariant matching functions:

  • Normalized Cross Correlation (NCC)
  • Zero-mean Sum of Absolute Differences (ZSAD)
  • Census Transform
  • Sobel Operator

Matching Space (MS) Representation

For each pixel and disparity level, the network calculates matching costs and confidence measures (likelihoods) for these four functions. This results in an 8-channel feature vector per pixel-disparity pair, forming a 4D "Matching Volume."

Hybrid Architecture

This Matching Volume is processed by state-of-the-art 3D CNN backends, such as variants of GCNet and PSMNet. By abstracting the input into mathematically robust descriptors like the Census transform, the network is shielded from domain-specific appearance shifts and forced to learn the underlying structural relationships of the matching problem.

Experimental Evaluation

The authors evaluated MS-Nets across several diverse datasets including SceneFlow for initial training, and KITTI, Middlebury 2014, and ETH3D for cross-domain testing. They also assessed performance on "scarce data" subsets to evaluate stability under limited supervision.

Key Results and Discussion

MS-Nets demonstrate superior generalization capabilities. When trained on synthetic data and tested on the Middlebury dataset, MS-GCNet achieved an error rate (bad2-noc) of 18.52%, a substantial improvement over the 30.42% error rate of the standard RGB-based GCNet.

Qualitative Improvements

Qualitatively, MS-Nets excel at recovering fine details and handling large planar surfaces. Standard models often produce noisy or "bumpy" disparity maps in unseen domains, whereas MS-Nets maintain structural integrity.

Qualitative comparison of disparity estimation. (c) shows the noisy output of a standard network, while (d) demonstrates the smoother, more accurate results from MS-Net.

Robustness to Data Scarcity

A significant finding is the stability of MS-Nets under limited supervision. While standard networks yield unpredictable results when trained on small datasets, MS-Nets remain consistent because their input is already a highly structured representation of the geometry.

Conclusion and Future Work

While MS-Nets offer exceptional generalization, they face a slight "specialization tax" when tested on the same domain used for training, as they cannot exploit domain-specific RGB cues. Future research may explore learning a set of domain-invariant matching functions or making the selection process differentiable. Despite a small computational overhead for pre-processing (~289ms), MS-Nets provide a robust solution for real-world stereo vision applications.

Summary updated

Abstract

End-to-end deep networks represent the state of the art for stereo matching. While excelling on images framing environments similar to the training set, major drops in accuracy occur in unseen domains (e.g., when moving from synthetic to real scenes). In this paper we introduce a novel family of architectures, namely Matching-Space Networks (MS-Nets), with improved generalization properties. By replacing learning-based feature extraction from image RGB values with matching functions and confidence measures from conventional wisdom, we move the learning process from the color space to the Matching Space, avoiding over-specialization to domain specific features. Extensive experimental results on four real datasets highlight that our proposal leads to superior generalization to unseen environments over conventional deep architectures, keeping accuracy on the source domain almost unaltered. Our code is available at https://qithub.com/ccj5351/MS-Nets.
BibTeX
@inproceedings{cai2020matching,
  author = {Changjiang Cai and Matteo Poggi and Stefano Mattoccia and Philippos Mordohai},
  title = {{Matching-space Stereo Networks for Cross-domain Generalization}},
  booktitle = {3DV},
  year = {2020},
  url = {https://openreview.net/forum?id=lgCCGDCl1og},
}
Figure