Digging Into Uncertainty-based Pseudo-label for Robust Stereo Matching

Zhelun Shen, Xibin Song, Yuchao Dai, Dingfu Zhou, Zhibo Rao, Liangjun Zhang

Introduction

Stereo matching—the process of estimating depth by comparing a pair of offset images—is a cornerstone of computer vision, powering technologies from autonomous vehicles to robotic systems. However, deep learning models for stereo matching often suffer from "fragility," performing exceptionally well on training data but failing when deployed in new environments. This is primarily due to domain shift, where models struggle with unfamiliar visual conditions and disparity distributions.

In their paper, "Digging Into Uncertainty-based Pseudo-label for Robust Stereo Matching," researchers from Baidu Research and Northwestern Polytechnical University present UCFNet, a framework that leverages self-estimated uncertainty to bridge this gap and achieve robust generalization.

The Challenge of Domain Shift

The core challenge in stereo matching is adapting to diverse environments. A model trained on synthetic, brightly lit scenes often fails to handle real-world complexities like highway shadows or high-resolution indoor textures. Beyond visual differences, datasets vary significantly in their disparity distributions; for example, indoor scenes may have depth ranges many times larger than outdoor ones. Most current models use a fixed search range, making them incapable of perceiving depths outside their pre-defined window without retraining.

Methodology: The UCFNet Framework

UCFNet (Uncertainty-based Cascade and Fused cost volume) addresses these issues through three primary innovations:

  1. Fused Cost Volume: Instead of a single scale, UCFNet fuses multiple low-resolution cost volumes to provide a robust initial guess that captures both fine edges and broad context.
  2. Uncertainty-based Cascade: While most models use a fixed coarse-to-fine strategy, UCFNet uses pixel-level uncertainty to adaptively adjust its search space. If the model is confident, it narrows its search; if uncertain, it widens the space to avoid missing the correct depth.
  3. Pseudo-label Generation for Adaptation: To learn a new domain without ground truth, the model generates its own labels. To ensure accuracy, it uses a dual-level uncertainty filtering mechanism (pixel-level and area-level) to discard noisy predictions, retaining only the most reliable points for training.
The generated pseudo-labels provide a reliable basis for domain adaptation, filtering out noisy predictions through uncertainty estimation.

Key Insights

  • Uncertainty as a Functional Tool: A key innovation is treating uncertainty not just as an error metric, but as an architectural driver. By letting uncertainty dictate the search range, the model adaptively prunes unlikely correspondences.
  • Reliability Over Density: For domain adaptation, the authors found that a sparse set of highly accurate labels is far more valuable than a dense set of noisy ones. Their dual-level filtering is essential for extracting high-quality training signals from the model's own predictions.
  • Cross-Task Versatility: The pseudo-labels generated are of such high quality that they can even be used to train monocular depth networks, achieving results comparable to models trained with actual sensor-based ground truth.

Results and Performance

UCFNet's robustness was validated by earning 1st place in the stereo task of the Robust Vision Challenge 2020. On the KITTI 2015 dataset, the uncertainty-based pseudo-labeling approach reduced the error rate from 5.2% to 3.1% without using any real ground-truth data from that domain. Qualitatively, the model is significantly better at filling in textureless areas (such as car surfaces) and regions where traditional sensors often fail (such as the sky).

Generalization on KITTI 2015. UCFNet significantly improves performance in textureless foreground areas (red box) and unlabeled background areas (green box) compared to standard pre-trained versions.

Conclusion and Future Work

UCFNet demonstrates that self-estimated uncertainty is a powerful tool for creating robust, adaptable stereo matching systems. While highly effective, the authors note that extreme situations—such as dark, featureless tunnels—can still cause initial matching signals to fail. Future work will explore integrating more advanced domain-invariant features and making the adaptation process real-time for on-the-fly environment learning.

Summary updated

Abstract

Due to the domain differences and unbalanced disparity distribution across multiple datasets, current stereo matching approaches are commonly limited to a specific dataset and generalize poorly to others. Such domain shift issue is usually addressed by substantial adaptation on costly target-domain ground-truth data, which cannot be easily obtained in practical settings. In this paper, we propose to dig into uncertainty estimation for robust stereo matching. Specifically, to balance the disparity distribution, we employ a pixel-level uncertainty estimation to adaptively adjust the next stage disparity searching space, in this way driving the network progressively prune out the space of unlikely correspondences. Then, to solve the limited ground truth data, an uncertainty-based pseudo-label is proposed to adapt the pre-trained model to the new domain, where pixel-level and area-level uncertainty estimation are proposed to filter out the high-uncertainty pixels of predicted disparity maps and generate sparse while reliable pseudo-labels to align the domain gap. Experimentally, our method shows strong cross-domain, adapt, and joint generalization and obtains \textbf{1st} place on the stereo task of Robust Vision Challenge 2020. Additionally, our uncertainty-based pseudo-labels can be extended to train monocular depth estimation networks in an unsupervised way and even achieves comparable performance with the supervised methods. The code will be available at https://github.com/gallenszl/UCFNet.
BibTeX
@article{shen2023digging,
  author = {Zhelun Shen and Xibin Song and Yuchao Dai and Dingfu Zhou and Zhibo Rao and Liangjun Zhang},
  title = {{Digging Into Uncertainty-based Pseudo-label for Robust Stereo Matching}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2023},
  doi = {10.1109/tpami.2023.3300976},
}
Figure