Masked Representation Learning for Domain Generalized Stereo Matching

Zhibo Rao, Bangshu Xiong, Mingyi He, Yuchao Dai, Renjie He, Zhelun Shen, Xing LiCVPR 2023 · DOI

Stereo matching is a fundamental task in computer vision, aiming to estimate depth by finding correspondences between a pair of rectified images. While deep learning models have achieved remarkable accuracy on synthetic benchmarks, they often struggle to generalize to real-world environments. This "domain gap" is typically caused by models overfitting to the specific textures and lighting conditions of synthetic training data.

The Challenge: Volatility and Overfitting

A significant but often overlooked issue in domain generalization is performance volatility. Even when a model eventually reaches a good error rate on a target domain, its performance can fluctuate wildly across training epochs. Since ground truth is unavailable for unseen real-world data, selecting the "best" epoch for deployment becomes a gamble. This instability suggests that models are learning fragile, texture-based features rather than robust geometric structures.

Proposed Method: Masked Representation Learning (MRL)

To address these issues, the authors propose a "pseudo multi-task" framework that incorporates Masked Representation Learning (MRL). This strategy forces the network to learn structural context by reconstructing missing parts of the input.

The proposed MRL framework integrates a reconstruction pipeline with the standard stereo matching pipeline.

The framework consists of several key steps:

  1. Random Masking: A portion of the left image (typically 15-25%) is randomly masked out. The right image remains intact to provide a reference for the matching task.
  2. Feature Extraction: Both images pass through a shared feature extractor. The network must use the surrounding context of the masked regions and the information from the right image to understand the scene.
  3. Reconstruction Task: A lightweight decoder is added to the feature extraction stage. Its sole purpose during training is to reconstruct the original, unmasked left image from the extracted features.
  4. Joint Optimization: The model is trained using a combination of the standard matching loss (for disparity estimation) and a reconstruction loss (MSE).

Crucially, the reconstruction decoder is discarded during inference. This means the model benefits from more robust features without any additional computational cost or latency during deployment.

Results: Stability and Generalization

The MRL strategy was evaluated by training on the synthetic SceneFlow dataset and testing on four real-world datasets: KITTI 2012, KITTI 2015, ETH3D, and Middlebury.

Performance volatility across training epochs for CFNet and LacGwcNet on various datasets.

The results demonstrate two primary benefits:

  • Enhanced Stability: As shown in the figure above, the "Masked" versions of CFNet and LacGwcNet exhibit significantly flatter performance curves across epochs compared to their baselines. For instance, on KITTI 2012, the variance in End-Point Error (EPE) for LacGwcNet dropped from 0.32 to a mere 0.03.
  • Improved Accuracy: The models achieved lower EPE across all target domains. On the challenging Middlebury dataset, Mask-CFNet reduced the EPE from 2.45 to 1.86, a substantial gain in zero-shot generalization.

Key Insights

The success of MRL lies in its ability to prioritize structure over texture. By forcing the model to "fill in the blanks," the network learns to rely on global geometric patterns that are consistent across different domains, rather than local pixel-level textures that vary wildly between synthetic and real-world scenes. Furthermore, the authors found that a relatively low masking ratio is optimal for stereo matching, as the task still requires significant local detail to find precise correspondences.

In conclusion, Masked Representation Learning provides a simple, effective, and "free" way to improve the robustness and reliability of stereo matching models for real-world applications.

Summary updated

BibTeX
@inproceedings{rao2023masked,
  author = {Zhibo Rao and Bangshu Xiong and Mingyi He and Yuchao Dai and Renjie He and Zhelun Shen and Xing Li},
  title = {{Masked Representation Learning for Domain Generalized Stereo Matching}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.00526},
}

Loading related content...

Figure