MaDis-Stereo: Enhanced Stereo Matching via Distilled Masked Image Modeling

Jihye Ahn, Hyesong Choi, Soomin Kim, Dongbo MinIEEE Access 2024 · DOI · OpenReview

Abstract

In stereo matching, Convolutional Neural Networks (CNNs) have traditionally served as the predominant architectures due to their inherent locality inductive bias. Although Transformer-based models have been studied recently, their performance often lags behind CNNs due to data scarcity. We propose MaDis-Stereo, which enhances locality inductive bias by leveraging Masked Image Modeling (MIM). Given randomly masked stereo images, our method performs both image reconstruction and depth prediction. To address training instability, we utilize an auxiliary teacher network updated via Exponential Moving Average (EMA). Teacher knowledge is distilled to the student model via pseudo-supervisory signals. MaDis-Stereo achieves state-of-the-art results on the ETH3D benchmark and competitive performance on KITTI 2015.

Introduction

Stereo matching—the process of calculating depth by comparing two slightly offset images—has long been dominated by CNNs. CNNs possess an inherent "locality inductive bias," looking at neighboring pixels to find matching features. While Transformers excel at modeling global relationships, they are data-hungry and lack this built-in local focus. Because high-quality ground truth depth data is scarce, Transformers struggle to learn the fine-grained local correspondences required for accurate depth perception.

Methodology

The MaDis-Stereo framework integrates self-supervision to impart a locality inductive bias into Transformer architectures.

Masked Image Modeling (MIM)

During training, the model receives stereo pairs where approximately 40% of image patches are randomly masked. The network is tasked with simultaneously reconstructing missing pixels and predicting the disparity map. This dual task forces the model to rely on surrounding pixels to fill in "holes," effectively mimicking the local behavior of CNNs while retaining Transformer global modeling capabilities.

Student-Teacher Distillation

Reconstructing images while matching stereo features can lead to training instability. To mitigate this, we employ a Student-Teacher framework. The Student network processes masked images, while a Teacher network—updated via Exponential Moving Average (EMA) of the Student's weights—processes the original, unmasked images. The Teacher provides dense pseudo-labels that guide the Student in regions where ground truth data is missing or sparse.

The MaDis-Stereo architecture featuring a Student-Teacher framework with Masked Image Modeling (MIM) and Exponential Moving Average (EMA) distillation.

Experimental Results

The model was evaluated on the KITTI 2015 and ETH3D benchmarks, utilizing weights from CroCo-Stereo pre-trained on synthetic and real datasets.

Performance Benchmarks

MaDis-Stereo achieved state-of-the-art performance on major leaderboards. On KITTI 2015, it reached a D1-all error of 1.57%, outperforming previous leaders like IGEV-Stereo. On ETH3D, it significantly improved the "[email protected]" metric, dropping from a 3.51% baseline to 2.96%. Qualitative results demonstrate sharper edges and fewer artifacts in complex regions compared to predecessors.

Analysis of Locality Bias

To verify the impact of MIM, we analyzed the "attention distance" across Transformer layers. As shown in Figure 3, MaDis-Stereo’s attention heads focus on significantly closer pixels compared to the standard CroCo-Stereo baseline, confirming the successful integration of locality inductive bias.

Comparison of attention distance across layers, demonstrating that MaDis-Stereo achieves a stronger locality inductive bias compared to the baseline CroCo-Stereo.

Conclusion

MaDis-Stereo demonstrates that integrating Masked Image Modeling into Transformer training can overcome data scarcity issues in stereo matching by enhancing locality inductive bias. While the Student-Teacher architecture increases training overhead, the gains in stability and accuracy are substantial. Future work may explore saliency-based masking strategies and knowledge distillation into real-time architectures for autonomous driving applications.

Summary updated

Abstract

In stereo matching, CNNs have traditionally served as the predominant architectures. Although Transformer-based stereo models have been studied recently, their performance still lags behind CNN-based stereo models due to the inherent data scarcity issue in the stereo matching task. In this paper, we propose Masked Image Modeling Distilled Stereo matching model, termed MaDis-Stereo, that enhances locality inductive bias by leveraging Masked Image Modeling (MIM) in training Transformer-based stereo model. Given randomly masked stereo images as inputs, our method attempts to conduct both image reconstruction and depth prediction tasks. While this strategy is beneficial to resolving the data scarcity issue, the dual challenge of reconstructing masked tokens and subsequently performing stereo matching poses significant challenges, particularly in terms of training stability. To address this, we propose to use an auxiliary network (teacher), updated via Exponential Moving Average (EMA), along with the original stereo model (student), where teacher predictions serve as pseudo supervisory signals to effectively distill knowledge into the student model. State-of-the-arts performance is achieved with the proposed method on several stereo matching such as ETH3D and KITTI 2015. Additionally, to demonstrate that our model effectively leverages locality inductive bias, we provide the attention distance measurement.
BibTeX
@inproceedings{ahn2024madis,
  author = {Jihye Ahn and Hyesong Choi and Soomin Kim and Dongbo Min},
  title = {{MaDis-Stereo: Enhanced Stereo Matching via Distilled Masked Image Modeling}},
  booktitle = {IEEE Access},
  year = {2024},
  doi = {10.1109/access.2025.3528022},
}
Figure