SRH-Net: Stacked Recurrent Hourglass Network for Stereo Matching

Hongzhi Du, Yanyan Li, Yanbiao Sun, Jigui Zhu, Federico Tombari

Abstract

Stereo matching is a fundamental task in computer vision, yet modern high-performance models often suffer from prohibitive memory requirements due to the use of 3D convolutions. We present SRH-Net, a memory-efficient architecture that replaces resource-heavy 3D cost aggregation with a novel Stacked Recurrent Hourglass (SRH) module. By processing cost maps sequentially and leveraging multi-scale context through an encoder-decoder structure, SRH-Net reduces GPU memory consumption by up to 56.1% compared to state-of-the-art models like PSMNet, while maintaining competitive accuracy on high-resolution benchmarks.

Introduction

Reconstructing 3D depth from stereo image pairs requires calculating disparity—the horizontal shift between corresponding pixels. Current deep learning approaches typically choose between two extremes: fast but less accurate 2D encoder-decoders, or highly accurate but memory-intensive 3D convolutional networks (3DConv). 3DConv methods build a 4D cost volume that scales cubically with resolution, often leading to "Out of Memory" errors on high-resolution data or mobile hardware. SRH-Net bridges this gap by introducing a sequential aggregation strategy that scales linearly with disparity range.

Methodology: The Stacked Recurrent Hourglass

The core innovation of SRH-Net is the Stacked Recurrent Hourglass (SRH) module. Unlike traditional methods that process the entire 4D cost volume simultaneously, SRH-Net treats the disparity dimension as a sequence, processing 2D cost maps one by one.

Overview of the SRH-Net architecture, illustrating the sequential flow from feature extraction to the SRH aggregation block and final disparity regression.

Standard recurrent units like Gated Recurrent Units (GRUs) often lack the receptive field necessary to handle "textureless" regions. To solve this, SRH-Net embeds GRUs within an hourglass (encoder-decoder) structure. The encoder downsamples cost maps to capture global context, while the decoder upsamples them to maintain spatial resolution. By stacking two such hourglasses with skip connections, the network preserves fine-grained details while effectively "looking around" to find matching anchors in difficult areas.

Experimental Results

SRH-Net was evaluated on the Scene Flow, KITTI 2012/2015, and Middlebury 2014 datasets. The results demonstrate that memory efficiency does not require a sacrifice in performance.

Memory and Scalability

On the KITTI benchmark, SRH-Net required only 2.04 GB of GPU memory, a 56.1% reduction from PSMNet's 4.65 GB. This efficiency allowed the model to process high-resolution images (up to 2960 × 2000 pixels) that typically crash standard 3DConv architectures.

Accuracy and Robustness

In textureless regions, the SRH module outperformed standard stacked GRU methods by 38% in terms of End-Point Error (EPE). This confirms that the multi-scale hourglass design is critical for resolving ambiguities where local pixel information is insufficient.

Qualitative comparison on the Scene Flow dataset. SRH-Net (c) significantly improves upon the standard Stacked GRU baseline (b), producing results that closely align with the Ground Truth (d).

Conclusion

SRH-Net offers a viable path for deploying high-quality depth sensing on memory-constrained devices. By decoupling the disparity dimension through recurrent hourglasses, it achieves the robustness of 3D convolutional methods with the efficiency of 2D architectures. Future work will focus on optimizing inference speed to match its superior memory profile.

Summary updated

BibTeX
@article{du2021srh,
  author = {Hongzhi Du and Yanyan Li and Yanbiao Sun and Jigui Zhu and Federico Tombari},
  title = {{SRH-Net: Stacked Recurrent Hourglass Network for Stereo Matching}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2021},
  url = {https://openreview.net/forum?id=PDCk362e0v},
}
Figure