StereoNet: Guided Hierarchical Refinement for Real-Time Edge-Aware Depth Prediction

Sameh Khamis, Sean Ryan Fanello, Christoph Rhemann, Adarsh Kowdle, Julien P. C. Valentin, Shahram Izadi

TL;DR

This paper introduces StereoNet, a novel deep learning architecture for real-time stereo depth prediction that achieves state-of-the-art accuracy at 60fps. By leveraging a low-resolution cost volume and a hierarchical edge-aware refinement process, StereoNet significantly reduces computational cost while maintaining high sub-pixel precision, making it suitable for demanding applications like augmented reality.

What Problem Does This Paper Address?

Estimating depth from stereo images is a fundamental problem in computer vision with applications ranging from 3D reconstruction to augmented reality (AR) and autonomous driving. Traditional stereo matching methods, while capable of high accuracy, are often computationally expensive, making them unsuitable for real-time applications, especially on resource-constrained devices. Deep learning approaches have shown promise, but often require significant processing power and memory, or struggle with fine details and edge preservation. The challenge is to achieve both high accuracy (especially sub-pixel precision) and real-time performance while preserving fine geometric details and sharp edges.

Methods

StereoNet employs a two-stage, end-to-end deep learning architecture:

  • Coarse Prediction: Cost Volume Filtering:
    • A Siamese network with shared weights extracts features from the left and right input images. These features are downsampled aggressively (e.g., by a factor of 8) to create a large receptive field, which helps in handling textureless regions.
  • A low-resolution cost volume is constructed by matching these features. The key insight here is that this low-resolution volume, despite its reduced spatial resolution, contains sufficient information for high sub-pixel disparity precision.
  • This cost volume is then filtered using 3D convolutions to aggregate spatial and disparity context.
  • A differentiable "soft arg min" operation is used to select the disparity from the filtered cost volume, allowing gradients to flow back through the network during training. This is a crucial departure from traditional hard arg min, which is non-differentiable.
  • Hierarchical Refinement: Edge-Aware Upsampling:
    • To recover fine details and preserve edges, a hierarchical refinement process is used.
  • The coarse disparity map is upsampled (using bilinear upsampling to avoid checkerboard artifacts) and combined with the original color image.
  • Compact, context-aware pixel-to-pixel refinement networks (using residual blocks and atrous convolutions) learn to predict a disparity residual. This residual is added to the upsampled coarse prediction.
  • This refinement is applied hierarchically, allowing the network to progressively recover high-frequency details and edge information, guided by the color image.

Datasets

The paper uses the following datasets:

  • Scene Flow: A large synthetic stereo dataset used for initial training and evaluation.
  • KITTI 2012 & KITTI 2015: Real-world datasets used for fine-tuning and evaluating the model's performance on more challenging, less controlled scenarios.

Key Insights and Ideas

  • Low-Resolution Cost Volume for High Precision: The most significant insight is that a low-resolution cost volume can encode enough information to achieve very high sub-pixel disparity precision, comparable to or exceeding traditional methods that operate at full resolution. This is a major computational bottleneck reduction.
  • Sub-pixel Precision is Key: Deep learning architectures can achieve an order of magnitude higher sub-pixel matching precision than traditional stereo methods. This high precision is what enables the use of low-resolution cost volumes.
  • Hierarchical Edge-Aware Refinement: Recovering fine details and preserving edges is achieved through a learned, hierarchical upsampling process that uses the color image as a guide, mimicking the behavior of joint bilateral upsampling but in a learned, data-driven manner.
  • Differentiable Arg Min is Crucial: Using a differentiable soft arg min allows the entire network, including the cost volume computation, to be trained end-to-end.
  • Over-parameterization in Prior Work: The authors suggest that previous deep stereo methods might have been over-parameterized, especially in their cost volume stages, and that their more compact approach achieves comparable or better results with significantly less computation.

Key Results

  • Real-time Performance: StereoNet achieves 60fps on an NVIDIA Titan X GPU, even for 720p images.
  • High Sub-pixel Precision: The method demonstrates a sub-pixel precision of 0.03 pixels, which is an order of magnitude better than traditional stereo approaches and significantly lower than previous deep learning methods. This translates to very small errors even at long distances.
  • State-of-the-Art Accuracy: StereoNet achieves state-of-the-art end-point error (EPE) on the Scene Flow dataset (1.1 EPE for the best multi-scale architecture).
  • Competitive Results on KITTI: While not always reaching the absolute state-of-the-art on KITTI, StereoNet provides compelling results with significantly faster runtimes compared to other top-performing methods. For instance, it achieves a runtime of 0.015s per stereo pair on KITTI, compared to 0.9s or 67s for other methods.
  • Edge Preservation: Qualitative results show that StereoNet effectively preserves edges and recovers fine details, even when starting from a low-resolution cost volume.

Limitations and Future Work

  • Supervised Training Data: A primary limitation is the reliance on supervised training data. The authors note that their method reaches state-of-the-art results when sufficient examples are available.
  • Future Work: To address the data limitation, future work will involve combining supervised and self-supervised learning techniques to augment the training set. The authors also suggest that their architecture's simplicity and effectiveness could lead to further advancements in overcoming current limitations.

Summary updated

Abstract

This paper presents StereoNet, the first end-to-end deep architecture for real-time stereo matching that runs at 60fps on an NVidia Titan X, producing high-quality, edge-preserved, quantization-free disparity maps. A key insight of this paper is that the network achieves a sub-pixel matching precision than is a magnitude higher than those of traditional stereo matching approaches. This allows us to achieve real-time performance by using a very low resolution cost volume that encodes all the information needed to achieve high disparity precision. Spatial precision is achieved by employing a learned edge-aware upsampling function. Our model uses a Siamese network to extract features from the left and right image. A first estimate of the disparity is computed in a very low resolution cost volume, then hierarchically the model re-introduces high-frequency details through a learned upsampling function that uses compact pixel-to-pixel refinement networks. Leveraging color input as a guide, this function is capable of producing high-quality edge-aware output. We achieve compelling results on multiple benchmarks, showing how the proposed method offers extreme flexibility at an acceptable computational budget.
BibTeX
@inproceedings{khamis2018stereonet,
  author = {Sameh Khamis and Sean Ryan Fanello and Christoph Rhemann and Adarsh Kowdle and Julien P. C. Valentin and Shahram Izadi},
  title = {{StereoNet: Guided Hierarchical Refinement for Real-Time Edge-Aware Depth Prediction}},
  booktitle = {European Conference on Computer Vision},
  year = {2018},
  doi = {10.1007/978-3-030-01267-0\_35},
}
Figure