Introduction
Stereo matching—the process of estimating depth from a pair of rectified images—is a fundamental task for autonomous systems. While deep learning has significantly advanced the state-of-the-art, these models typically require massive amounts of labeled data. Because high-quality, dense ground truth is difficult to obtain in the real world, researchers rely heavily on synthetic datasets.
However, models trained on synthetic data often suffer from a "sim-to-real" gap. This paper identifies shortcut learning as a primary cause: instead of learning the underlying geometry of a scene, networks latch onto "transient" features unique to synthetic environments, such as specific color distributions or texture patterns. When deployed in real-world scenarios where these shortcuts no longer exist, model performance collapses.
Methodology: Information-Theoretic Shortcut Avoidance
To combat shortcut learning, the authors propose ITSA (Information-Theoretic Shortcut Avoidance). This model-agnostic training strategy is built on the Information Bottleneck (IB) principle, which seeks a feature representation that is maximally predictive of the target while being minimally complex.
Standard IB, however, does not inherently distinguish between robust geometric features and brittle shortcuts. ITSA addresses this by incorporating Fisher Information into the loss function. Fisher Information measures the sensitivity of the model's internal features to small input perturbations. Since shortcuts are typically transient and highly sensitive, minimizing Fisher Information effectively "squeezes out" these unreliable cues.
Key components of the framework include:
- Shortcut Perturbation (SCP): A mechanism that identifies and perturbs input pixels that most strongly influence the model's internal features.
- Fisher Information Loss ($\mathcal{L}_{FI}$): A computationally efficient first-order approximation that allows the model to minimize sensitivity without the overhead of second-order derivatives.
Experimental Evaluation
The authors evaluated ITSA across several benchmark datasets to test its generalization capabilities. Models were trained on the synthetic Scene Flow dataset and tested on real-world datasets including KITTI (2012 & 2015), Middlebury, and ETH3D. To further test robustness, they utilized DrivingStereo (rain/fog) and Oxford Robotcar (night-time) to simulate challenging environmental conditions.
Key Results and Generalization
The results demonstrate that avoiding shortcuts leads to significantly better real-world performance. For example, an ITSA-optimized PSMNet reduced the D1 error on KITTI 2015 from 29.3% to 5.8% compared to the synthetic baseline. On the high-resolution Middlebury set, the error dropped from 60.4% to 28.4%.
Notably, ITSA-trained models often outperformed standard models that had been specifically fine-tuned on real-world data when tested in anomalous conditions. In rainy and night-time scenarios, ITSA-trained models achieved an average error of 6.46%, surpassing the 8.28% error of models fine-tuned on clean, sunny real-world data. This suggests that the features learned via ITSA are more fundamentally robust and "common-sense" oriented.
Conclusion and Future Work
ITSA provides a powerful, model-agnostic framework for improving the domain generalization of stereo matching networks. By mathematically penalizing sensitivity to transient input features, the network is forced to prioritize stable geometric cues. While the method significantly improves performance, challenges remain in extreme conditions like specular reflections or near-total darkness. Future research will focus on refining these physical edge cases and improving the efficiency of the Fisher Information approximation for larger architectures.
Summary updated