Stereo Matching by Self-supervision of Multiscopic Vision

Weihao Yuan, Yazhan Zhang, Bingkun Wu, Siyu Zhu, Ping Tan, Michael Yu Wang, Qifeng Chen

TL;DR

This paper introduces a novel self-supervised learning framework for stereo matching that leverages multiscopic vision (three or more aligned cameras) instead of standard stereo pairs during training. By exploiting the strict geometric constraints of parallel cameras, the authors introduce cross-photometric and uncertainty-aware mutual-supervision losses to train a highly accurate depth estimation network without any ground-truth data. The resulting model generalizes exceptionally well to unseen real-world datasets and enables on-the-fly online fine-tuning.

What Problem Does This Paper Address?

Accurate depth estimation is critical for robotics, autonomous driving, and 3D reconstruction. While deep learning has revolutionized stereo matching, supervised models require massive amounts of ground-truth depth data. Collecting this data in the real world (e.g., via LiDAR) is expensive, sparse, and difficult to align. Consequently, models trained on limited datasets tend to overfit and fail catastrophically when deployed in unseen environments.

To bypass the need for ground truth, researchers have turned to unsupervised or self-supervised learning, typically using standard stereo image pairs to train networks via image reconstruction (warping the left image to match the right). However, these methods struggle heavily with occlusions—pixels visible in one camera but hidden in the other—and lack robust geometric constraints. This paper asks: can we use multiscopic images (multiple cameras aligned on the same axis) during training to provide stronger constraints and solve the occlusion problem in self-supervised depth estimation?

Methods

The authors propose an end-to-end self-supervised framework built on a modified PSMNet backbone. While the network only requires a standard stereo pair (two images) during inference, it uses three aligned images (left, center, right) during training.

The training process relies on three key losses:

  1. Cross Photometric Loss: Because the cameras share the same baseline, the pixel disparity from the center to the left image should be identical to the disparity from the center to the right image. The network uses predicted disparities to warp the surrounding images to the center view (and vice versa) and penalizes visual differences.
  2. Uncertainty-Aware Mutual-Supervision Loss: This is the core innovation to handle occlusions. The network predicts an uncertainty map alongside the disparity map. If the center-left matching is highly uncertain (e.g., an object is blocking the background in the left view), but the center-right matching is confident, the network uses the confident right-side prediction as a pseudo-label to supervise and correct the uncertain left-side prediction.
  3. Smoothness Loss: A regularization term that encourages adjacent pixels with similar colors to have similar depth disparities.

Datasets

Because no large-scale multiscopic dataset existed, the authors built a new one:

  • Synthetic Multiscopic Dataset: 1,200 scenes rendered using 3D engines (Habitat-sim, SceneNet RGB-D) populated with real-world scanned assets (Matterport3D, Gibson) and virtual objects (ShapeNet). This dataset includes ground-truth depth for evaluation.
  • Real-World Multiscopic Dataset: 100 scenes captured using custom hardware rigs (a 3-camera array and a moving-eye slider system) to ensure perfectly parallel, co-planar optical axes.
  • Standard Benchmarks: The model was also evaluated on KITTI 2015 (outdoor driving) and Cityscapes to test its ability to generalize to entirely unseen domains.

Key Insights and Ideas

  • A third camera solves the two-camera occlusion problem: The biggest "aha" moment is how the framework handles occluded pixels. In standard stereo, an occluded pixel breaks the photometric loss. By adding a third camera on the opposite side, a pixel hidden from the left camera is almost certainly visible to the right camera. The uncertainty-aware mutual supervision dynamically routes the "good" geometry to fix the "bad" geometry.
  • Self-supervision beats supervised learning for generalization: When trained only on synthetic indoor multiscopic data, the self-supervised model generalized remarkably well to outdoor KITTI driving scenes. A fully supervised model trained on the exact same synthetic data failed on KITTI. Supervised models memorize their training domain; self-supervised models are forced to learn the underlying physics and geometry of stereo matching.
  • Free online fine-tuning: Because the method requires zero ground-truth depth, the model can continuously learn and adapt in the wild. If a robot enters a new environment, it can use its own multiscopic camera feed to fine-tune its depth perception on the fly.

Key Results

  • KITTI 2015: Despite never seeing outdoor driving data during training, the model achieved a D1-all error of 3.82% on the training set and 4.43% on the test set. This outperformed previous unsupervised methods (e.g., UnOS at 5.943% / 6.67%) by a massive margin.
  • Synthetic Test Set: The model achieved an End-Point Error (EPE) of 1.19, nearly matching the fully supervised PSMNet (1.07) and drastically outperforming previous unsupervised baselines (2.16).
  • Ablation on Occlusions: Adding the uncertainty-aware mutual supervision dropped the D1 error on occluded pixels from 38.71% (using just basic cross-camera loss) down to 10.87%, proving its effectiveness at solving the occlusion challenge.

Limitations and Future Work

  • Hardware Constraints: Capturing multiscopic training data requires specialized, perfectly aligned camera rigs (parallel, co-planar, exact same baselines), which is much harder to build and calibrate than a standard stereo camera.
  • Computational Efficiency: The network relies on a heavy 3D convolution backbone (PSMNet), resulting in a forward pass time of ~0.5 seconds per image pair. This is not fast enough for high-speed real-time robotics, though the authors note the backbone could be swapped for a lighter architecture.
  • Future Directions: The authors suggest that training directly on outdoor multiscopic data (rather than just synthetic indoor data) would push KITTI benchmark errors even lower. They also note that multiscopic setups could be leveraged for other self-supervised tasks like super-resolution and novel view synthesis.

Summary updated

Abstract

Self-supervised learning for depth estimation possesses several advantages over supervised learning. The benefits of no need for ground-truth depth, online fine-tuning, and better generalization with unlimited data attract researchers to seek self-supervised solutions. In this work, we propose a new self-supervised framework for stereo matching utilizing multiple images captured at aligned camera positions. A cross photometric loss, an uncertainty-aware mutual-supervision loss, and a new smoothness loss are introduced to optimize the network in learning disparity maps end-to-end without ground-truth depth information. To train this framework, we build a new multiscopic dataset consisting of synthetic images rendered by 3D engines and real images captured by real cameras. After being trained with only the synthetic images, our network can perform well in unseen outdoor scenes. Our experiment shows that our model obtains better disparity maps than previous unsupervised methods on the KITTI dataset and is comparable to supervised methods when generalized to unseen data. Our source code and dataset are available at https://sites.google.com/view/multiscopic.
BibTeX
@inproceedings{yuan2021stereo,
  author = {Weihao Yuan and Yazhan Zhang and Bingkun Wu and Siyu Zhu and Ping Tan and Michael Yu Wang and Qifeng Chen},
  title = {{Stereo Matching by Self-supervision of Multiscopic Vision}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2021},
  doi = {10.1109/iros51168.2021.9636616},
}

Loading related content...

Figure