Abstract
Unlike other vision tasks where Transformer-based approaches are becoming increasingly common, stereo depth estimation is still dominated by convolution-based approaches. This is mainly due to the limited availability of real-world ground truth for stereo matching, which is a limiting factor in improving the performance of Transformer-based stereo approaches. In this paper, we propose UniTT-Stereo, a method to maximize the potential of Transformer-based stereo architectures by unifying self-supervised learning used for pre-training with stereo matching framework based on supervised learning. To be specific, we explore the effectiveness of reconstructing features of masked portions in an input image and at the same time predicting corresponding points in another image from the perspective of locality inductive bias, which is crucial in training models with limited training data. Moreover, to address these challenging tasks of reconstruction-and-prediction, we present a new strategy to vary a masking ratio when training the stereo model with stereo-tailored losses. State-of-the-art performance of UniTT-Stereo is validated on various benchmarks such as ETH3D, KITTI 2012, and KITTI 2015 datasets. Lastly, to investigate the advantages of the proposed approach, we provide a frequency analysis of feature maps and the analysis of locality inductive bias based on attention maps.
Introduction
Stereo depth estimation—the process of calculating distance by comparing two slightly offset images—has traditionally been dominated by Convolutional Neural Networks (CNNs). While Transformers have revolutionized many areas of computer vision, they have struggled to achieve the same dominance in stereo matching. This lag is primarily due to two factors: the scarcity of high-quality real-world ground truth data (often requiring expensive LiDAR setups) and the inherent lack of "locality inductive bias" in Transformers.
Unlike CNNs, which naturally focus on local pixel relationships, Transformers are global by nature. In stereo matching, where identifying fine-grained local patterns between images is critical, this lack of bias can lead to blurry depth maps. UniTT-Stereo addresses this by introducing a unified training framework that bridges self-supervised masked image modeling and supervised stereo matching, effectively injecting the necessary locality bias into the Transformer architecture.
Methodology
The UniTT-Stereo architecture utilizes a Siamese Vision Transformer (ViT) encoder paired with a specialized "Inter Image Information Exchange" (I3E) transformer decoder.
Unified Training Task
During training, the model is presented with a masked left image and a full right image. It is tasked with simultaneously reconstructing the missing patches of the left image (a self-supervised task) and estimating the disparity for every pixel (a supervised task). This dual objective forces the model to learn local context to fill in the blanks, which directly benefits the precision of the stereo matching process.
Variable Ratio Masking
Rather than using a fixed masking percentage, the authors implement a Variable Ratio Masking strategy. By sampling masking ratios from a truncated normal distribution, the model is exposed to varying levels of difficulty. High masking ratios encourage the model to learn broad contextual relationships for reconstruction, while lower ratios allow it to refine the fine-grained details essential for sharp depth boundaries.
Synergistic Loss Functions
The framework is optimized through a combination of three distinct losses:
- Disparity Loss: A standard supervised loss comparing predictions against ground truth.
- Image Reconstruction Loss: Ensures the model can accurately fill in masked image regions.
- Feature Consistency Loss: Encourages alignment between the features of both images by warping right-image features to the left using ground truth disparity.
Key Insights and Analysis
Locality via Reconstruction
The core innovation of UniTT-Stereo is using reconstruction as a proxy for teaching locality. To reconstruct a missing patch, the Transformer must attend to its immediate neighbors, effectively "baking in" the locality bias that CNNs possess by design. This leads to more accurate correspondences between the stereo pair.
High-Frequency Feature Enhancement
Fourier analysis of the feature maps reveals that this unified training significantly amplifies high-frequency components. As shown in the frequency analysis, UniTT-Stereo captures more detail in the higher frequency spectrum compared to "plain" Transformer models. In practice, this translates to the model's superior ability to identify sharp edges and fine structures, such as tree branches or fence posts, which are often lost in traditional approaches.
Experimental Results
UniTT-Stereo achieves state-of-the-art performance across several major benchmarks, demonstrating its robustness and accuracy.
- ETH3D: The model outperformed previous leaders like CREStereo and IGEV-Stereo, particularly in the "[email protected]" metric, achieving a score of 0.83 for non-occluded pixels.
- KITTI 2015: It reached a D1-all error rate of 1.47%, surpassing established CNN and Transformer hybrid models.
- Qualitative Sharpness: Visual comparisons on the KITTI dataset show that UniTT-Stereo produces significantly cleaner boundaries around objects, avoiding the common issue of "bleeding" or blurring at depth discontinuities.
Conclusion
UniTT-Stereo demonstrates that the potential of Transformers in stereo matching can be fully realized by unifying self-supervised reconstruction with supervised learning. By addressing the locality bias problem through a novel masking and training strategy, the model achieves superior accuracy and sharper depth maps. While the model currently faces challenges with very large maximum disparities due to its tiling-based inference, future work focusing on larger tile sizes or refined masking strategies promises to further enhance its generalization and performance in complex real-world environments.
Summary updated