Stereo Matching by Self-supervision of Multiscopic Vision

Weihao Yuan, Yazhan Zhang, Bingkun Wu, Siyu Zhu, Ping Tan, Michael Yu Wang, Qifeng Chen

Abstract

Self-supervised learning for depth estimation possesses several advantages over supervised learning. The benefits of no need for ground-truth depth, online fine-tuning, and better generalization with unlimited data attract researchers to seek self-supervised solutions. In this work, we propose a new self-supervised framework for stereo matching utilizing multiple images captured at aligned camera positions. A cross photometric loss, an uncertainty-aware mutual-supervision loss, and a new smoothness loss are introduced to optimize the network in learning disparity maps end-to-end without ground-truth depth information. To train this framework, we build a new multiscopic dataset consisting of synthetic images rendered by 3D engines and real images captured by real cameras. After being trained with only the synthetic images, our network can perform well in unseen outdoor scenes. Our experiment shows that our model obtains better disparity maps than previous unsupervised methods on the KITTI dataset and is comparable to supervised methods when generalized to unseen data.

Introduction

Accurate depth estimation is a cornerstone of robotics, autonomous driving, and 3D reconstruction. While deep learning has significantly advanced stereo matching, traditional supervised models are bottlenecked by the need for massive amounts of ground-truth depth data. Collecting such data in the real world—often via LiDAR—is expensive, sparse, and difficult to align precisely with image sensors. Consequently, models trained on limited supervised datasets often overfit, failing when deployed in novel environments.

To address these challenges, researchers have turned to self-supervised learning. Standard self-supervised stereo typically uses image pairs to train networks via image reconstruction (warping one image to match the other). However, these methods frequently struggle with occlusions—pixels visible in one camera but hidden in the other—and lack robust geometric constraints. This paper introduces a framework that leverages multiscopic vision (three or more aligned cameras) during training to provide stronger geometric constraints and effectively solve the occlusion problem.

The Multiscopic Framework

The proposed end-to-end self-supervised framework is built upon a modified PSMNet backbone. A key advantage is that while the network utilizes three or more aligned images (e.g., left, center, and right) during the training phase to enforce geometric consistency, it only requires a standard stereo pair for inference.

Self-supervised multiscopic training framework utilizing three aligned images to compute cross-photometric and mutual-supervision losses.

As shown in the architecture diagram above, the training process is driven by three primary loss functions:

  1. Cross Photometric Loss: Leveraging the shared baseline of the cameras, the network ensures that pixel disparity from the center to the left image matches the disparity from the center to the right. It uses predicted disparities to warp surrounding images to the center view, penalizing visual discrepancies.
  2. Uncertainty-Aware Mutual-Supervision Loss: This core innovation addresses occlusions. The network predicts an uncertainty map alongside the disparity map. If a match is highly uncertain (e.g., due to an occlusion in the left view) but the corresponding match in another view is confident, the network uses the confident prediction as a pseudo-label to correct the uncertain one.
  3. Smoothness Loss: This regularization term encourages adjacent pixels with similar colors to maintain similar depth disparities, ensuring spatial consistency.

Datasets and Generalization

Because no large-scale multiscopic dataset was previously available, the authors curated a new, diverse collection of images to facilitate training and evaluation.

The proposed dataset includes synthetic scenes with ground-truth disparity (a, b) and real-world captures (c) to facilitate robust self-supervised learning.
  • Synthetic Multiscopic Dataset: Comprising 1,200 scenes rendered using 3D engines (Habitat-sim, SceneNet RGB-D), this dataset utilizes real-world scanned assets (Matterport3D, Gibson) and virtual objects (ShapeNet). It includes ground-truth depth for rigorous evaluation.
  • Real-World Multiscopic Dataset: 100 scenes were captured using custom hardware rigs, including a 3-camera array and a moving-eye slider system, to ensure perfectly parallel and co-planar optical axes.
  • Standard Benchmarks: To test domain generalization, the model was evaluated on the KITTI 2015 and Cityscapes datasets without any prior exposure to outdoor driving data during training.

Key Insights and Results

Solving the Occlusion Problem

In standard stereo, occluded pixels break the photometric loss. By introducing a third camera on the opposite side, pixels hidden from one view are typically visible in the other. The uncertainty-aware mutual supervision dynamically routes reliable geometric information to fix errors in occluded regions. Ablation studies showed that this mechanism dropped the D1 error on occluded pixels from 38.71% to just 10.87%.

Superior Generalization

A major finding is that self-supervision can outperform supervised learning in terms of generalization. When trained only on synthetic indoor data, the self-supervised model performed remarkably well on outdoor KITTI scenes, achieving a D1-all error of 4.43% on the test set. This significantly outperformed previous unsupervised methods like UnOS (6.67%). In contrast, fully supervised models trained on the same synthetic data often fail to adapt to the real-world physics of unseen domains.

Online Adaptation

Because the method requires no ground-truth depth, it enables "free" online fine-tuning. A system can continuously adapt its depth perception in real-time as it encounters new environments, simply by processing its own multiscopic camera feed.

Limitations and Future Work

While powerful, the approach requires specialized, perfectly aligned camera hardware for training data collection. Additionally, the current implementation uses a heavy 3D convolution backbone (PSMNet), leading to a processing time of approximately 0.5 seconds per pair. Future work may involve adopting lighter architectures for real-time robotics and exploring multiscopic setups for tasks like super-resolution and novel view synthesis.

Summary updated

Abstract

Self-supervised learning for depth estimation possesses several advantages over supervised learning. The benefits of no need for ground-truth depth, online fine-tuning, and better generalization with unlimited data attract researchers to seek self-supervised solutions. In this work, we propose a new self-supervised framework for stereo matching utilizing multiple images captured at aligned camera positions. A cross photometric loss, an uncertainty-aware mutual-supervision loss, and a new smoothness loss are introduced to optimize the network in learning disparity maps end-to-end without ground-truth depth information. To train this framework, we build a new multiscopic dataset consisting of synthetic images rendered by 3D engines and real images captured by real cameras. After being trained with only the synthetic images, our network can perform well in unseen outdoor scenes. Our experiment shows that our model obtains better disparity maps than previous unsupervised methods on the KITTI dataset and is comparable to supervised methods when generalized to unseen data. Our source code and dataset are available at https://sites.google.com/view/multiscopic.
BibTeX
@inproceedings{yuan2021stereo,
  author = {Weihao Yuan and Yazhan Zhang and Bingkun Wu and Siyu Zhu and Ping Tan and Michael Yu Wang and Qifeng Chen},
  title = {{Stereo Matching by Self-supervision of Multiscopic Vision}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2021},
  doi = {10.1109/iros51168.2021.9636616},
}
Figure