Unsupervised Learning of Stereo Matching

Chao Zhou, Hong Zhang, Xiaoyong Shen, Jiaya JiaICCV 2017 · OpenReview

Abstract

Convolutional neural networks showed the ability in stereo matching cost learning. Recent approaches learned parameters from public datasets that have ground truth disparity maps. Due to the difficulty of labeling ground truth depth, usable data for system training is rather limited, making it difficult to apply the system to real applications. In this paper, we present a framework for learning stereo matching costs without human supervision. Our method updates network parameters in an iterative manner. It starts with a randomly initialized network. Left-right check is adopted to guide the training. Suitable matching is then picked and used as training data in following iterations. Our system finally converges to a stable state and performs even comparably with other supervised methods.

Introduction

Stereo matching—the process of estimating depth by comparing two slightly offset images—is a cornerstone of computer vision. While Convolutional Neural Networks (CNNs) have significantly improved this field, they are notoriously data-hungry. The primary bottleneck is the availability of ground truth data. Obtaining accurate depth maps for real-world scenes requires expensive hardware like LIDAR, which is often noisy or limited to specific environments. Synthetic datasets exist, but models trained on "perfect" CGI often fail when faced with the messy complexity of the real world. This paper addresses the challenge of training a high-performing stereo matching model using only raw image pairs, without any pre-existing labels.

Methodology

Network Architecture

The proposed network consists of three main components designed to handle matching and structural consistency:

  1. Cost-Volume Branch (CVB): Learns to extract features and compute the initial matching cost between patches.
  2. Image-Feature Branch (IFB): Extracts structural and color information from the original images.
  3. Joint Filtering Branch (JFB): A novel component that combines matching costs with image features to perform "cost aggregation," effectively smoothing results based on the image's actual edges.
The detailed network architecture showing the Cost-volume Branch, Image Feature Branch, and Joint Filter.

To make the disparity prediction differentiable, the authors employ a Soft Argmax operator instead of a hard "winner-take-all" approach, allowing for end-to-end training.

Iterative Unsupervised Learning

The core of the framework is an iterative self-supervised loop. The network starts with random weights and progressively refines its predictions through a "virtuous cycle."

The iterative training framework showing the loop from input to disparity, confidence map generation, and new training data selection across iterations.

The process relies on a Left-Right Consistency Check: the system calculates disparity from both left-to-right and right-to-left. If the values for a specific pixel do not match, that pixel is flagged as unreliable (likely due to occlusion). Only "confidential" pixels—those passing the check—are used as pseudo-labels for the next training iteration. As the network improves, the area of confident pixels expands, providing higher quality training data until convergence.

Key Insights

  • Self-Labeling via Consistency: Using geometric constraints as a supervisor allows the model to learn without human labels. If the world is consistent, the depth calculated from the left eye should match the depth from the right eye.
  • Feature-Based Matching: Unlike previous methods relying on pixel-intensity constancy, this model matches learned features, making it robust to lighting changes and shadows.
  • Integrated Cost Aggregation: The Joint Filtering Branch learns to smooth data by observing the original image's structure, replacing traditional hand-crafted filters.

Results and Evaluation

The model was primarily evaluated on the KITTI 2015 stereo benchmark, using 160 pairs for training (without ground truth) and 40 for validation.

  • Performance: The model achieved a "Non-Occluded" (NOC) error rate of 9.23% (>2 pixels), outperforming the supervised MC-CNN (13.20%) and previous unsupervised state-of-the-art methods like USCNN (11.17%).
  • Efficiency: Testing takes approximately 0.39 seconds per image pair on a TitanX GPU, making it fast enough for practical applications.
  • Convergence: The iterative process typically reaches a stable, high-accuracy state within about 50 iterations.

Conclusion

This framework demonstrates that high-quality stereo matching can be achieved without human supervision. By leveraging iterative self-labeling and learned feature matching, the system rivals supervised models. Future work may extend this unsupervised approach to other matching problems like optical flow or object tracking, and further address challenges in textureless regions and occluded areas.

Summary updated

Abstract

Convolutional neural networks showed the ability in stereo matching cost learning. Recent approaches learned parameters from public datasets that have ground truth disparity maps. Due to the difficulty of labeling ground truth depth, usable data for system training is rather limited, making it difficult to apply the system to real applications. In this paper, we present a framework for learning stereo matching costs without human supervision. Our method updates network parameters in an iterative manner. It starts with a randomly initialized network. Left-right check is adopted to guide the training. Suitable matching is then picked and used as training data in following iterations. Our system finally converges to a stable state and performs even comparably with other supervised methods.
BibTeX
@inproceedings{zhou2017unsupervised,
  author = {Chao Zhou and Hong Zhang and Xiaoyong Shen and Jiaya Jia},
  title = {{Unsupervised Learning of Stereo Matching}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2017},
  url = {https://openreview.net/forum?id=HJ4SLbf_bH},
}
Figure