Abstract
Deep learning has significantly advanced stereo vision, yet models trained on synthetic data often fail to generalize to real-world scenes due to the domain gap. This paper introduces a method to enhance generalization by incorporating geometric priors—specifically piecewise smoothness and occlusion reasoning—into the training process as auxiliary loss functions. These modules are utilized only during training and discarded during inference, resulting in substantial accuracy gains on real-world datasets like Middlebury 2014 without increasing the model's runtime or parameter count.
Introduction
Estimating depth from stereo image pairs is a fundamental task in computer vision. While deep learning models have achieved state-of-the-art performance, they are notoriously data-hungry. Collecting dense, high-quality ground truth for real-world scenes is challenging, often requiring expensive LiDAR setups that struggle with dynamic objects. Consequently, researchers rely on large-scale synthetic datasets like FlyingThings3D. However, models often overfit to synthetic textures and lighting, leading to poor performance in real-world applications. This work addresses this "domain gap" by making networks "geometry-aware," forcing them to respect universal physical laws that remain consistent across both synthetic and real environments.
Methodology
The authors propose two back-propagatable training modules that act as geometric "teachers" for the stereo network. These modules calculate gradients to update the network's weights during training but are not required for deployment.
Gradient-Domain Smoothness Prior
Traditional smoothness constraints often cause "bleeding" across object boundaries. To mitigate this, the authors employ Pixel-Adaptive Convolutions (PAC). PACs use features from the original image to guide the filtering of disparity gradients ($D_x, D_y$). This approach encourages the model to represent surfaces as slanted planes or smooth curves, aligning more closely with the physical structure of real-world objects rather than treating them as flat, front-facing cards.
Occlusion Reasoning
Occlusions—regions visible to only one camera—frequently cause errors in stereo matching. The authors derive a deterministic relationship between disparity and occlusion maps based on rectified camera geometry. By implementing a "soft" version of this logic using a steep Sigmoid function, the network learns to predict disparities and occlusions that are geometrically consistent with each other.
Experimental Results
The approach was evaluated by training on synthetic datasets (FlyingThings3D and Falling Things) and testing on the real-world Middlebury 2014 dataset, which serves as a gold standard for generalization testing.
Quantitative and Qualitative Performance
Integrating these geometric priors into the PSM-Net architecture led to a dramatic improvement in accuracy. On the Middlebury dataset, the Mean Absolute Error (MAE) decreased from 5.37 to 3.21.
Qualitative analysis reveals that the gradient-domain smoothness prior effectively reduces noise in textureless regions, such as walls and floors, where standard networks typically struggle. Furthermore, the occlusion module helps the network maintain consistency in challenging regions near object boundaries.
Discussion and Key Insights
A primary advantage of this method is the "free lunch" it provides: because the auxiliary modules are only used during training, the final model maintains its original inference speed and memory footprint. The results suggest that while textures vary between domains, geometry is a universal invariant. By shifting focus to second-order priors (gradients), the authors demonstrate that the structural change in depth is often more informative than absolute depth values for generalization.
Conclusion
This work demonstrates that incorporating geometric consistency into the training of deep stereo networks significantly bridges the gap between synthetic and real-world performance. While the occlusion module showed varying impact across different architectures, the overall framework provides a robust way to improve model reliability. Future work may explore extending these priors to related tasks such as monocular depth estimation and optical flow, where geometric constraints are equally fundamental.
Summary updated