Improving Deep Stereo Network Generalization with Geometric Priors

Jialiang Wang, Varun Jampani, Deqing Sun, Charles T. Loop, Stan Birchfield, Jan KautzarXiv 2020 · arXiv

Abstract

Deep learning has significantly advanced stereo vision, yet models trained on synthetic data often fail to generalize to real-world scenes due to the domain gap. This paper introduces a method to enhance generalization by incorporating geometric priors—specifically piecewise smoothness and occlusion reasoning—into the training process as auxiliary loss functions. These modules are utilized only during training and discarded during inference, resulting in substantial accuracy gains on real-world datasets like Middlebury 2014 without increasing the model's runtime or parameter count.

Introduction

Estimating depth from stereo image pairs is a fundamental task in computer vision. While deep learning models have achieved state-of-the-art performance, they are notoriously data-hungry. Collecting dense, high-quality ground truth for real-world scenes is challenging, often requiring expensive LiDAR setups that struggle with dynamic objects. Consequently, researchers rely on large-scale synthetic datasets like FlyingThings3D. However, models often overfit to synthetic textures and lighting, leading to poor performance in real-world applications. This work addresses this "domain gap" by making networks "geometry-aware," forcing them to respect universal physical laws that remain consistent across both synthetic and real environments.

Methodology

The authors propose two back-propagatable training modules that act as geometric "teachers" for the stereo network. These modules calculate gradients to update the network's weights during training but are not required for deployment.

Gradient-Domain Smoothness Prior

Traditional smoothness constraints often cause "bleeding" across object boundaries. To mitigate this, the authors employ Pixel-Adaptive Convolutions (PAC). PACs use features from the original image to guide the filtering of disparity gradients ($D_x, D_y$). This approach encourages the model to represent surfaces as slanted planes or smooth curves, aligning more closely with the physical structure of real-world objects rather than treating them as flat, front-facing cards.

Occlusion Reasoning

Occlusions—regions visible to only one camera—frequently cause errors in stereo matching. The authors derive a deterministic relationship between disparity and occlusion maps based on rectified camera geometry. By implementing a "soft" version of this logic using a steep Sigmoid function, the network learns to predict disparities and occlusions that are geometrically consistent with each other.

Experimental Results

The approach was evaluated by training on synthetic datasets (FlyingThings3D and Falling Things) and testing on the real-world Middlebury 2014 dataset, which serves as a gold standard for generalization testing.

Quantitative and Qualitative Performance

Integrating these geometric priors into the PSM-Net architecture led to a dramatic improvement in accuracy. On the Middlebury dataset, the Mean Absolute Error (MAE) decreased from 5.37 to 3.21.

Qualitative comparison of disparity estimation on the Middlebury dataset. The proposed geometric priors significantly reduce noise and improve boundary definition compared to the baseline PSM-Net.

Qualitative analysis reveals that the gradient-domain smoothness prior effectively reduces noise in textureless regions, such as walls and floors, where standard networks typically struggle. Furthermore, the occlusion module helps the network maintain consistency in challenging regions near object boundaries.

Discussion and Key Insights

A primary advantage of this method is the "free lunch" it provides: because the auxiliary modules are only used during training, the final model maintains its original inference speed and memory footprint. The results suggest that while textures vary between domains, geometry is a universal invariant. By shifting focus to second-order priors (gradients), the authors demonstrate that the structural change in depth is often more informative than absolute depth values for generalization.

Conclusion

This work demonstrates that incorporating geometric consistency into the training of deep stereo networks significantly bridges the gap between synthetic and real-world performance. While the occlusion module showed varying impact across different architectures, the overall framework provides a robust way to improve model reliability. Future work may explore extending these priors to related tasks such as monocular depth estimation and optical flow, where geometric constraints are equally fundamental.

Summary updated

Abstract

End-to-end deep learning methods have advanced stereo vision in recent years and obtained excellent results when the training and test data are similar. However, large datasets of diverse real-world scenes with dense ground truth are difficult to obtain and currently not publicly available to the research community. As a result, many algorithms rely on small real-world datasets of similar scenes or synthetic datasets, but end-to-end algorithms trained on such datasets often generalize poorly to different images that arise in real-world applications. As a step towards addressing this problem, we propose to incorporate prior knowledge of scene geometry into an end-to-end stereo network to help networks generalize better. For a given network, we explicitly add a gradient-domain smoothness prior and occlusion reasoning into the network training, while the architecture remains unchanged during inference. Experimentally, we show consistent improvements if we train on synthetic datasets and test on the Middlebury (real images) dataset. Noticeably, we improve PSM-Net accuracy on Middlebury from 5.37 MAE to 3.21 MAE without sacrificing speed.
BibTeX
@misc{wang2020improving,
  author = {Jialiang Wang and Varun Jampani and Deqing Sun and Charles T. Loop and Stan Birchfield and Jan Kautz},
  title = {{Improving Deep Stereo Network Generalization with Geometric Priors}},
  year = {2020},
  eprint = {2008.11098},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2008.11098},
}
Figure