SRH-Net: Stacked Recurrent Hourglass Network for Stereo Matching

Hongzhi Du, Yanyan Li, Yanbiao Sun, Jigui Zhu, Federico Tombari

TL;DR

SRH-Net introduces a memory-efficient cost aggregation strategy for stereo matching that replaces resource-heavy 3D convolutions with a novel Stacked Recurrent Hourglass (SRH) module. By processing cost maps sequentially rather than as a massive 4D block, it reduces GPU memory consumption by up to 56.1% compared to state-of-the-art models like PSMNet while maintaining—and in some cases exceeding—accuracy on high-resolution benchmarks.

What Problem Does This Paper Address?

Stereo matching is the process of calculating "disparity" (the spatial shift between pixels in a left and right image) to reconstruct 3D depth. Modern deep learning approaches generally fall into two camps:

  1. 2D Encoder-Decoders: Fast and memory-efficient, but they often struggle with "textureless" areas (like a blank white wall) because they lack sufficient global context.
  2. 3D Convolutional Networks (3DConv): These build a 4D "cost volume" and use 3D filters to aggregate information. They are highly accurate but are notorious "memory hogs," often crashing on high-resolution images due to the cubic growth of computation requirements.

The authors aim to bridge this gap: achieving the robust performance of 3DConv methods without the massive hardware overhead, making high-quality depth sensing feasible for robots and mobile devices with limited GPU memory.

Methods

The core innovation is the Stacked Recurrent Hourglass (SRH) module used for cost aggregation.

  • Sequential Decoupling: Instead of feeding a 4D cubic cost volume into 3D filters, SRH-Net treats the disparity dimension as a sequence. It processes 2D cost maps one by one.
  • The SRH Module: Standard recurrent units (like GRUs) have a limited "receptive field"—they can't see the "big picture" of the image easily. To fix this, the authors embedded Gated Recurrent Units (GRUs) within an hourglass (encoder-decoder) structure.
  • The "encoder" part downsamples the cost map to capture multi-scale, global context.
  • The "decoder" part upsamples it back to the original resolution.
  • Stacking and Skip Connections: Two of these hourglasses are stacked together. Skip connections are used to pass fine-grained details from the early layers to the later ones, which is critical for maintaining sharp edges in the resulting depth map.
  • End-to-End Pipeline: The network starts with a 2D Siamese feature extractor (using Spatial Pyramid Pooling), followed by the SRH aggregation block, and ends with a "soft argmin" layer to regress the final disparity.

Datasets

The authors evaluated SRH-Net on several industry-standard benchmarks:

  • Scene Flow: A massive synthetic dataset (approx. 40k images) used for initial training and ablation studies.
  • KITTI 2012 & 2015: Real-world driving datasets captured with LiDAR. These are the "gold standard" for evaluating how models handle outdoor lighting and complex geometry.
  • Middlebury 2014: A dataset featuring high-resolution indoor scenes, used here to demonstrate the model's scalability to large images.

Key Insights and Ideas

  • Receptive Field is King: The paper demonstrates that simply using recurrent units (like the Stacked GRU in RMVSNet) isn't enough for stereo matching. Without the multi-scale processing of the hourglass structure, recurrent models fail in textureless regions because they can't "look around" far enough to find matching anchors.
  • Linear vs. Cubic Scaling: By moving from 3D convolutions to sequential 2D recurrent processing, the memory cost scales much more gracefully with image resolution.
  • Efficiency without Sacrifice: The most surprising finding is that this memory-saving approach doesn't result in a "lite" version of the results. On the KITTI 2015 benchmark, SRH-Net actually outperformed the 3DConv-based PSMNet in several metrics while using half the memory.

Key Results

  • Memory Savings: SRH-Net required only 2.04 GB of GPU memory on KITTI, compared to 4.65 GB for PSMNet—a 56.1% reduction.
  • Accuracy: On the Scene Flow dataset, SRH-Net achieved an End-Point Error (EPE) of 1.09 pixels, matching the performance of heavy 3DConv models.
  • Textureless Robustness: Compared to standard stacked GRU methods, the SRH module improved the average EPE by 38%, proving the value of the multi-scale hourglass design.
  • High-Res Scalability: The authors successfully ran the model on images as large as 2960 × 2000 pixels, a feat that typically causes 3DConv models to fail with "Out of Memory" errors.

Limitations and Future Work

While SRH-Net is highly efficient, the paper notes a few areas for improvement:

  • Inference Speed: Although memory usage is low, processing cost maps sequentially can be slower than parallel 3D convolutions on high-end hardware (SRH-Net takes 0.5s vs PSMNet's 0.4s on KITTI).
  • Fixed Disparity Range: Like many current models, the maximum disparity is still somewhat "baked into" the architecture, though the recurrent nature makes it easier to adjust than 3DConv counterparts.
  • Future Directions: The authors suggest that further optimizing the recurrent units or exploring different encoder-decoder architectures within the SRH module could lead to even better generalization across different types of scenes.

Summary updated

Abstract

The cost aggregation strategy shows a crucialrole in learning-based stereo matching tasks, where 3D convolutional filters obtain state of the art but require intensive computation resources, while 2D operations need less GPU memory but are sensitive to domain shift. In this letter, we decouple the 4D cubic cost volume used by 3D convolutional filters into sequential cost maps along the direction of disparity instead of dealing with it at once by exploiting a recurrent cost aggregation strategy. Furthermore, a novel recurrent module, Stacked Recurrent Hourglass (SRH), is proposed to process each cost map. Our hourglass network is constructed based on Gated Recurrent Units (GRUs) and down/upsampling layers, which provides GRUs larger receptive fields. Then two hourglass networks are stacked together, while multi-scale information is processed by skip connections to enhance the performance of the pipeline in textureless areas. The proposed architecture is implemented in an end-to-end pipeline and evaluated on public datasets, which reduces GPU memory consumption by up to 56.1% compared with PSMNet using stacked hourglass 3D CNNs without the degradation of accuracy. Then, we further demonstrate the scalability of the proposed method on several high-resolution pairs, while previously learned approaches often fail due to the memory constraint. The code is released at https://github.com/hongzhidu/SRHNet.
BibTeX
@article{du2021srh,
  author = {Hongzhi Du and Yanyan Li and Yanbiao Sun and Jigui Zhu and Federico Tombari},
  title = {{SRH-Net: Stacked Recurrent Hourglass Network for Stereo Matching}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2021},
  url = {https://openreview.net/forum?id=PDCk362e0v},
}
Figure