ITSA: An Information-Theoretic Approach to Automatic Shortcut Avoidance and Domain Generalization in Stereo Matching Networks

Weiqin Chuah, Ruwan B. Tennakoon, Reza Hoseinnezhad, Alireza Bab-Hadiashar, David SuterCVPR 2022 · DOI · arXiv

Introduction

Stereo matching—the process of estimating depth from a pair of rectified images—is a fundamental task for autonomous systems. While deep learning has significantly advanced the state-of-the-art, these models typically require massive amounts of labeled data. Because high-quality, dense ground truth is difficult to obtain in the real world, researchers rely heavily on synthetic datasets.

However, models trained on synthetic data often suffer from a "sim-to-real" gap. This paper identifies shortcut learning as a primary cause: instead of learning the underlying geometry of a scene, networks latch onto "transient" features unique to synthetic environments, such as specific color distributions or texture patterns. When deployed in real-world scenarios where these shortcuts no longer exist, model performance collapses.

Methodology: Information-Theoretic Shortcut Avoidance

To combat shortcut learning, the authors propose ITSA (Information-Theoretic Shortcut Avoidance). This model-agnostic training strategy is built on the Information Bottleneck (IB) principle, which seeks a feature representation that is maximally predictive of the target while being minimally complex.

The ITSA training pipeline. It introduces Shortcut Perturbation (SCP) to identify sensitive input regions and applies a Fisher Information Loss (L_FI) to regularize the feature representation z.

Standard IB, however, does not inherently distinguish between robust geometric features and brittle shortcuts. ITSA addresses this by incorporating Fisher Information into the loss function. Fisher Information measures the sensitivity of the model's internal features to small input perturbations. Since shortcuts are typically transient and highly sensitive, minimizing Fisher Information effectively "squeezes out" these unreliable cues.

Key components of the framework include:

  • Shortcut Perturbation (SCP): A mechanism that identifies and perturbs input pixels that most strongly influence the model's internal features.
  • Fisher Information Loss ($\mathcal{L}_{FI}$): A computationally efficient first-order approximation that allows the model to minimize sensitivity without the overhead of second-order derivatives.

Experimental Evaluation

The authors evaluated ITSA across several benchmark datasets to test its generalization capabilities. Models were trained on the synthetic Scene Flow dataset and tested on real-world datasets including KITTI (2012 & 2015), Middlebury, and ETH3D. To further test robustness, they utilized DrivingStereo (rain/fog) and Oxford Robotcar (night-time) to simulate challenging environmental conditions.

Key Results and Generalization

The results demonstrate that avoiding shortcuts leads to significantly better real-world performance. For example, an ITSA-optimized PSMNet reduced the D1 error on KITTI 2015 from 29.3% to 5.8% compared to the synthetic baseline. On the high-resolution Middlebury set, the error dropped from 60.4% to 28.4%.

Qualitative comparison across diverse datasets (KITTI-15, DrivingStereo, Oxford, Middlebury). ITSA-trained models (Row IV) show significantly cleaner and more accurate depth maps compared to baselines.

Notably, ITSA-trained models often outperformed standard models that had been specifically fine-tuned on real-world data when tested in anomalous conditions. In rainy and night-time scenarios, ITSA-trained models achieved an average error of 6.46%, surpassing the 8.28% error of models fine-tuned on clean, sunny real-world data. This suggests that the features learned via ITSA are more fundamentally robust and "common-sense" oriented.

Conclusion and Future Work

ITSA provides a powerful, model-agnostic framework for improving the domain generalization of stereo matching networks. By mathematically penalizing sensitivity to transient input features, the network is forced to prioritize stable geometric cues. While the method significantly improves performance, challenges remain in extreme conditions like specular reflections or near-total darkness. Future research will focus on refining these physical edge cases and improving the efficiency of the Fisher Information approximation for larger architectures.

Summary updated

BibTeX
@inproceedings{chuah2022itsa,
  author = {Weiqin Chuah and Ruwan B. Tennakoon and Reza Hoseinnezhad and Alireza Bab-Hadiashar and David Suter},
  title = {{ITSA: An Information-Theoretic Approach to Automatic Shortcut Avoidance and Domain Generalization in Stereo Matching Networks}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  doi = {10.1109/cvpr52688.2022.01268},
}
Figure