UniTT-Stereo: Unified Training of Transformer for Enhanced Stereo Matching

Soomin Kim, Hyesong Choi, Jihye Ahn, Dongbo Min

TL;DR

UniTT-Stereo introduces a unified training framework that bridges the gap between self-supervised masked image modeling and supervised stereo matching. By forcing a Transformer to simultaneously reconstruct masked image patches and predict depth, the authors successfully inject "locality inductive bias" into the model, leading to state-of-the-art accuracy and sharper object boundaries even with limited real-world training data.

What Problem Does This Paper Address?

Stereo depth estimation—calculating distance by comparing two slightly offset images—has long been the stronghold of Convolutional Neural Networks (CNNs). While Transformers have revolutionized other areas of computer vision, they’ve struggled to dominate stereo matching.

The core issue is twofold:

  1. Data Scarcity: High-quality, real-world ground truth for stereo (often requiring expensive LiDAR setups) is hard to come by.
  2. Inductive Bias: Transformers are "global" by nature. Unlike CNNs, which have a built-in "locality bias" (pixels near each other are likely related), Transformers need massive amounts of data to learn these local relationships from scratch.

In stereo matching, local patterns are everything. If a model can't precisely match small, local features between two images, the resulting depth map will be blurry or inaccurate.

Methods

The authors propose UniTT-Stereo (Unified Training of Transformer), which uses a Siamese Vision Transformer (ViT) encoder and a specialized "Inter Image Information Exchange" (I3E) decoder.

The UniTT-Stereo architecture unifies image reconstruction with disparity estimation.

The key innovations include:

  • Unified Training Task: During training, the model is given a "masked" left image and a full right image. It must simultaneously perform two tasks: reconstruct the missing parts of the left image (self-supervised) and estimate the disparity/depth for every pixel (supervised).
  • Variable Ratio Masking: Instead of a fixed masking percentage, they use a Variable Ratio Masking strategy (sampled from a truncated normal distribution). High masking ratios force the model to learn broad context for reconstruction, while low ratios allow it to focus on the fine-grained details necessary for sharp depth maps.
  • Synergistic Losses: The model is optimized using a combination of three losses:
  • Disparity Loss: Standard supervised loss against ground truth.
  • Image Reconstruction Loss: Ensures the model can accurately "fill in the blanks" of the masked image.
  • Feature Consistency Loss: Encourages the features extracted from both images to align correctly by warping the right image features to the left using ground truth disparity.

Datasets

The model was trained on a mix of synthetic and real datasets, including SceneFlow, CREStereo, ETH3D, Booster, and Middlebury.

Evaluation was performed on major benchmarks:

  • ETH3D: High-resolution indoor/outdoor scenes.
  • KITTI 2012 & 2015: Real-world driving sequences.
  • Middlebury: Challenging indoor scenes with complex geometries.

Key Insights and Ideas

  • Locality via Reconstruction: The "aha moment" is that the act of reconstructing masked patches acts as a proxy for teaching the Transformer locality. To fill in a missing patch, the model must look at its immediate neighbors, effectively "baking in" the locality bias that CNNs get for free.
  • High-Frequency Boost: Fourier analysis (Figure 3) reveals that this unified training significantly amplifies high-frequency components in the feature maps. In practical terms, this means the model is much better at identifying sharp edges and fine structures (like tree branches or fence posts) compared to "plain" Transformer models.
  • Attention Guidance: By visualizing attention maps (Figure 4), the authors show that combining reconstruction and disparity losses helps the model's "eyes" stay focused on the correct corresponding points in the other image, reducing "incorrect attention" on distant, unrelated pixels.
Fourier analysis shows UniTT-Stereo captures more high-frequency details, leading to sharper boundaries.

Key Results

UniTT-Stereo achieved state-of-the-art performance on the ETH3D leaderboard and delivered highly competitive results on the KITTI benchmarks.

  • ETH3D: It outperformed previous leaders like CREStereo and IGEV-Stereo, particularly in the "[email protected]" metric (percentage of pixels with error > 1px), achieving a score of 0.83 (non-occluded).
  • KITTI 2015: It reached a D1-all error rate of 1.47%, surpassing several established CNN and Transformer hybrids.
  • Qualitative Sharpness: As seen in Figure 5, the model produces noticeably cleaner boundaries around objects compared to competitors, which often suffer from "bleeding" or blurring at depth discontinuities.
Qualitative comparison on KITTI shows UniTT-Stereo (right) produces much sharper object boundaries.

Limitations and Future Work

The authors note a limitation in handling very large maximum disparities. Because the model uses a tiling-based inference strategy to handle high resolutions, it can struggle with correspondences that span across tile boundaries if the disparity is too high.

Future work may involve exploring larger tile sizes or increasing the overlap ratio during inference to better capture these long-range correspondences. Additionally, the authors suggest that further refining the masking strategy could continue to improve the model's ability to generalize to entirely unseen domains.

Summary updated

Abstract

Unlike other vision tasks where Transformer-based approaches are becoming increasingly common, stereo depth estimation is still dominated by convolution-based approaches. This is mainly due to the limited availability of real-world ground truth for stereo matching, which is a limiting factor in improving the performance of Transformer-based stereo approaches. In this paper, we propose UniTT-Stereo, a method to maximize the potential of Transformer-based stereo architectures by unifying self-supervised learning used for pre-training with stereo matching framework based on supervised learning. To be specific, we explore the effectiveness of reconstructing features of masked portions in an input image and at the same time predicting corresponding points in another image from the perspective of locality inductive bias, which is crucial in training models with limited training data. Moreover, to address these challenging tasks of reconstruction-and-prediction, we present a new strategy to vary a masking ratio when training the stereo model with stereo-tailored losses. State-of-the-art performance of UniTT-Stereo is validated on various benchmarks such as ETH3D, KITTI 2012, and KITTI 2015 datasets. Lastly, to investigate the advantages of the proposed approach, we provide a frequency analysis of feature maps and the analysis of locality inductive bias based on attention maps.
BibTeX
@article{kim2025unitt,
  author = {Soomin Kim and Hyesong Choi and Jihye Ahn and Dongbo Min},
  title = {{UniTT-Stereo: Unified Training of Transformer for Enhanced Stereo Matching}},
  journal = {IEEE Access},
  year = {2025},
  doi = {10.1109/access.2025.3633291},
}

Loading related content...

Figure