Abstract
In stereo matching, Convolutional Neural Networks (CNNs) have traditionally served as the predominant architectures due to their inherent locality inductive bias. Although Transformer-based models have been studied recently, their performance often lags behind CNNs due to data scarcity. We propose MaDis-Stereo, which enhances locality inductive bias by leveraging Masked Image Modeling (MIM). Given randomly masked stereo images, our method performs both image reconstruction and depth prediction. To address training instability, we utilize an auxiliary teacher network updated via Exponential Moving Average (EMA). Teacher knowledge is distilled to the student model via pseudo-supervisory signals. MaDis-Stereo achieves state-of-the-art results on the ETH3D benchmark and competitive performance on KITTI 2015.
Introduction
Stereo matching—the process of calculating depth by comparing two slightly offset images—has long been dominated by CNNs. CNNs possess an inherent "locality inductive bias," looking at neighboring pixels to find matching features. While Transformers excel at modeling global relationships, they are data-hungry and lack this built-in local focus. Because high-quality ground truth depth data is scarce, Transformers struggle to learn the fine-grained local correspondences required for accurate depth perception.
Methodology
The MaDis-Stereo framework integrates self-supervision to impart a locality inductive bias into Transformer architectures.
Masked Image Modeling (MIM)
During training, the model receives stereo pairs where approximately 40% of image patches are randomly masked. The network is tasked with simultaneously reconstructing missing pixels and predicting the disparity map. This dual task forces the model to rely on surrounding pixels to fill in "holes," effectively mimicking the local behavior of CNNs while retaining Transformer global modeling capabilities.
Student-Teacher Distillation
Reconstructing images while matching stereo features can lead to training instability. To mitigate this, we employ a Student-Teacher framework. The Student network processes masked images, while a Teacher network—updated via Exponential Moving Average (EMA) of the Student's weights—processes the original, unmasked images. The Teacher provides dense pseudo-labels that guide the Student in regions where ground truth data is missing or sparse.
Experimental Results
The model was evaluated on the KITTI 2015 and ETH3D benchmarks, utilizing weights from CroCo-Stereo pre-trained on synthetic and real datasets.
Performance Benchmarks
MaDis-Stereo achieved state-of-the-art performance on major leaderboards. On KITTI 2015, it reached a D1-all error of 1.57%, outperforming previous leaders like IGEV-Stereo. On ETH3D, it significantly improved the "[email protected]" metric, dropping from a 3.51% baseline to 2.96%. Qualitative results demonstrate sharper edges and fewer artifacts in complex regions compared to predecessors.
Analysis of Locality Bias
To verify the impact of MIM, we analyzed the "attention distance" across Transformer layers. As shown in Figure 3, MaDis-Stereo’s attention heads focus on significantly closer pixels compared to the standard CroCo-Stereo baseline, confirming the successful integration of locality inductive bias.
Conclusion
MaDis-Stereo demonstrates that integrating Masked Image Modeling into Transformer training can overcome data scarcity issues in stereo matching by enhancing locality inductive bias. While the Student-Teacher architecture increases training overhead, the gains in stability and accuracy are substantial. Future work may explore saliency-based masking strategies and knowledge distillation into real-time architectures for autonomous driving applications.
Summary updated