Multi-scale Iterative Residuals for Fast and Scalable Stereo Matching

Kumail Raza, René Schuster, Didier StrickerCSCS 2021 · OpenReview

TL;DR

This paper introduces an iterative multi-scale coarse-to-fine refinement (iCFR) framework that significantly speeds up and scales existing deep learning stereo matching models without sacrificing accuracy. By leveraging multi-scale warped features and estimating disparity residuals, they achieve real-time performance and remarkable scalability, even up to 6K resolution.

What Problem Does This Paper Address?

The field of deep learning for stereo matching has seen impressive accuracy gains, but a significant gap persists between highly accurate but slow "state-of-the-art" models and faster, real-time models. This gap hinders practical applications in areas like autonomous driving and robotics, where both accuracy and speed are crucial. Existing real-time methods often compromise too much on accuracy, while the most accurate models are computationally prohibitive for many applications. The authors aim to bridge this gap by making existing accurate models faster and more scalable.

Methods

The core contribution is the iterative coarse-to-fine refinement (iCFR) framework. This framework works by:

  • Multi-scale Feature Extraction: It utilizes features extracted at various resolutions from a shared backbone network (e.g., a stacked hourglass network).
  • Disparity Residuals: Instead of directly predicting the full disparity at each scale, the framework predicts residuals – the difference between the current estimated disparity and the true disparity. This is achieved by warping the features of the right image using the upsampled disparity from the previous, coarser scale.
  • Reduced Cost Volume: By working with residuals and at lower resolutions, the search range for disparities in the cost volume can be significantly reduced. This drastically cuts down computational cost and memory usage.
  • Iterative Refinement: The process is iterated across multiple scales, starting from a coarse estimate and progressively refining it with residual predictions.
  • Final Refinement Network: A dedicated network is used at the end to recover any precision lost during the multi-scale process, particularly at object boundaries and occlusions.
  • Adaptability: The iCFR framework is designed to be modular, allowing it to be plugged into existing stereo matching architectures (like GANet and AANet) by replacing their prediction heads.

A key technical detail is the use of a symmetric cost volume that allows for negative disparity hypotheses, which helps correct initial overestimations of disparity.

Datasets

  • Sceneflow: Used for training and evaluation. This is a large dataset of synthetic images with dense ground truth, previously introduced by Mayer et al. [17].
  • KITTI 2015: Used for evaluation. This dataset contains real-world outdoor images but with sparse ground truth. The authors also mention using a "dense pseudo-ground truth" derived from AANet [23] for fine-tuning on KITTI, which helps produce more consistent disparity maps in areas with no ground truth (like the sky).

Key Insights and Ideas

  • Disparity Residuals as a Key to Efficiency: The insight that predicting residuals rather than full disparities at each scale is crucial. This allows for much smaller search spaces and computational savings.
  • Scalability through Multi-Scale Processing: By breaking down the problem into smaller, iterative steps at different resolutions, the framework becomes highly scalable, fitting into smaller GPU memory footprints.
  • Modularity for Broad Impact: The iCFR framework's ability to be adopted by different existing stereo matching architectures means its benefits can be widely realized without needing to retrain entire networks from scratch.
  • Symmetric Cost Volume for Better Correction: Allowing negative disparity hypotheses in the cost volume is a clever way to ensure that initial coarse estimates can be effectively corrected by subsequent residual predictions.

Key Results

  • Speed: The FRSNet (their proposed model using iCFR) achieves a 49x faster inference time compared to GANet-deep. It's also significantly faster than other real-time methods, with runtimes as low as 60ms on a GTX 1080Ti.
  • Scalability: The model can scale to input resolutions as high as 6K while maintaining inference times under one second.
  • Memory and FLOPs: FRSNet uses 4x less memory and requires only 16% of the FLOPs compared to GANet-deep.
  • Accuracy: The FRSNet achieves comparable accuracy to state-of-the-art models like AANet+ and GANet. For example, it shows a 0.05 px improvement in EPE and a 31% better D1 score on KITTI compared to AANet.
  • Performance on KITTI: Achieves competitive accuracy on the KITTI benchmark, outperforming other real-time stereo methods.

Limitations and Future Work

  • Number of Scales: The authors note a limitation related to the number of scales that can be effectively used, which is tied to the relationship between the cost aggregation technique, the maximum disparity (Dmax), and the down-sampling factors (si), as described in Eq. 2.

  • Future Work:

    • Extending the architecture with other fast prediction heads to further boost performance.
  • Adapting the framework for edge computing devices due to its reduced FLOPs and real-time capabilities.

Summary updated

Abstract

Despite the remarkable progress of deep learning in stereo matching, there exists a gap in accuracy between real-time models and slower state-of-the-art models which are suitable for practical applications. This paper presents an iterative multi-scale coarse-to-fine refinement (iCFR) framework to bridge this gap by allowing it to adopt any stereo matching network to make it fast, more efficient and scalable while keeping comparable accuracy. To reduce the computational cost of matching, we use multi-scale warped features to estimate disparity residuals and push the disparity search range in the cost volume to a minimum limit. Finally, we apply a refinement network to recover the loss of precision which is inherent in multi-scale approaches. We test our iCFR framework by adopting the matching networks from state-of-the art GANet and AANet. The result is 49$\times$ faster inference time compared to GANetdeep and 4$\times$ less memory consumption, with comparable error. Our best performing network, which we call FRSNet is scalable even up to an input resolution of 6K on a GTX 1080Ti, with inference time still below one second and comparable accuracy to AANet+. It out-performs all real-time stereo methods and achieves competitive accuracy on the KITTI benchmark.
BibTeX
@inproceedings{raza2021multi,
  author = {Kumail Raza and René Schuster and Didier Stricker},
  title = {{Multi-scale Iterative Residuals for Fast and Scalable Stereo Matching}},
  booktitle = {CSCS},
  year = {2021},
  url = {https://openreview.net/forum?id=SxIrOYH6R3h},
}
Figure