StereoNet: Guided Hierarchical Refinement for Real-Time Edge-Aware Depth Prediction

Sameh Khamis, Sean Fanello, Christoph Rhemann, Adarsh Kowdle, Julien Valentin, Shahram Izadi

TL;DR

StereoNet is the first end-to-end deep learning architecture to achieve high-quality stereo matching at 60fps. By leveraging the extreme sub-pixel precision of deep networks, it computes depth at a very low resolution and then uses a learned, edge-aware hierarchical refinement process to recover sharp details.

What Problem Does This Paper Address?

Stereo matching—estimating depth by comparing two slightly offset images—is a fundamental task for AR/VR, autonomous driving, and 3D reconstruction. The "gold standard" approach involves building a cost volume, a 3D structure that stores the matching cost for every pixel at every possible depth (disparity).

The challenge is a classic trade-off: high-resolution cost volumes provide great detail but are computationally expensive and memory-intensive, often taking seconds to process a single frame. Conversely, low-resolution volumes are fast but produce "blurry" depth maps that miss thin structures and sharp edges. StereoNet aims to break this trade-off, providing real-time performance without sacrificing spatial precision.

Methods

The authors propose a two-stage architecture:

  1. Coarse Prediction (The "Siamese" Network): The model uses a Siamese network to extract features from the left and right images. Crucially, it aggressively downsamples the input (by a factor of 8x or 16x). It then builds a very small cost volume at this low resolution. To pick the best depth value in a way that allows the whole system to learn, they use a soft argmin function, which produces a continuous, sub-pixel disparity estimate.
  2. Hierarchical Refinement: To get back the lost detail, the model uses a series of refinement layers. This isn't just standard upsampling; it's a learned, edge-aware process. The network takes the low-res disparity map and the original high-res color image as a "guide." It learns to predict a "residual" (a small correction) to add to the disparity, effectively snapping the depth boundaries to the edges visible in the color image.
Fig. 1. Model architecture. A two stage approach is proposed: first we extract image features at a lower resolution using a Siamese network. We then build a cost volume at that resolution by matching the features along the scanlines, giving us a coarse disparity estimate. We finally refine the results hierarchically to recover small details and thin structures.

Datasets

The model was trained and evaluated on three primary benchmarks:

  • Scene Flow: A massive synthetic dataset used for initial end-to-end training.
  • KITTI 2012 & 2015: Real-world datasets captured from a moving car, used for fine-tuning and demonstrating performance in challenging outdoor environments.

Key Insights and Ideas

  • Deep Sub-pixel Precision: The "aha moment" of the paper is that deep networks are incredibly good at matching. While traditional methods hit a wall at roughly 1/10th of a pixel precision, StereoNet achieves 1/30th of a pixel.
  • Spatial vs. Disparity Resolution: Because the network is so precise at matching (disparity resolution), you can afford to lose spatial resolution (image size) early on. You can match at 1/16th resolution and still have enough information to reconstruct a high-quality full-resolution map.
  • Refinement as Inpainting: The authors argue that the refinement stage acts similarly to joint bilateral upsampling, using the color image to guide where depth should change abruptly.

Key Results

  • Speed: StereoNet runs at 60fps on an NVidia Titan X for 720p images, making it viable for real-time applications.
  • Efficiency: The unrefined model uses only 360k parameters and runs in 12ms, yet it outperforms much larger models (like GC-Net with 3.5M parameters) on certain benchmarks.
  • Accuracy: On the Scene Flow dataset, the multi-scale version achieves a state-of-the-art End Point Error (EPE) of 1.10.
  • Precision: The system achieves a sub-pixel precision of 0.03, which translates to an expected error of less than 5mm at a 3-meter distance using standard hardware.
Fig. 2. Hierarchical refinement results. The result at each stage (top row), starting with the cost volume output in the top left corner, is updated with the output of the corresponding refinement network (bottom row). The refinement network output expectedly dilates and erodes around the edges using the color input as guide. The groundtruth is shown in the lower right corner. The average endpoint error at each stage for this example is: 3.27, 2.34, 1.80, and 1.26 respectively. Zoom in for details.

Limitations and Future Work

The authors note that most remaining errors occur in occluded regions (where a pixel is visible in one camera but not the other) and on reflective surfaces. Since there is no true geometric match in these areas, the network has to "guess" the depth.

Future work involves moving toward self-supervised learning to reduce the dependence on perfectly labeled ground-truth data, which is difficult and expensive to capture in the real world.

Summary updated

Abstract

This paper presents StereoNet, the first end-to-end deep architecture for real-time stereo matching that runs at 60 fps on an NVidia Titan X, producing high-quality, edge-preserved, quantization-free disparity maps. A key insight of this paper is that the network achieves a sub-pixel matching precision than is a magnitude higher than those of traditional stereo matching approaches. This allows us to achieve real-time performance by using a very low resolution cost volume that encodes all the information needed to achieve high disparity precision. Spatial precision is achieved by employing a learned edge-aware upsampling function. Our model uses a Siamese network to extract features from the left and right image. A first estimate of the disparity is computed in a very low resolution cost volume, then hierarchically the model re-introduces high-frequency details through a learned upsampling function that uses compact pixel-to-pixel refinement networks. Leveraging color input as a guide, this function is capable of producing high-quality edge-aware output. We achieve compelling results on multiple benchmarks, showing how the proposed method offers extreme flexibility at an acceptable computational budget.
BibTeX
@inproceedings{khamis2018stereonet,
  author = {Sameh Khamis and Sean Fanello and Christoph Rhemann and Adarsh Kowdle and Julien Valentin and Shahram Izadi},
  title = {{StereoNet: Guided Hierarchical Refinement for Real-Time Edge-Aware Depth Prediction}},
  booktitle = {European Conference on Computer Vision},
  year = {2018},
  doi = {10.1007/978-3-030-01267-0\_35},
}
Figure