Cascade Residual Learning: A Two-Stage Convolutional Neural Network for Stereo Matching

Jiahao Pang, Wenxiu Sun, Jimmy S. J. Ren, Chengxi Yang, Qiong Yan

Abstract

Stereo matching remains a fundamental challenge in computer vision, particularly in ill-posed regions such as occlusions and textureless surfaces. This paper introduces Cascade Residual Learning (CRL), a novel two-stage framework that iteratively refines disparity maps. By employing a full-resolution initial estimation followed by multiscale residual learning, CRL achieves state-of-the-art accuracy on major benchmarks while maintaining competitive computational efficiency.

Introduction

Calculating depth from stereo image pairs is essential for applications ranging from autonomous driving to robotics. While deep learning has significantly advanced the field, standard Convolutional Neural Networks (CNNs) often struggle with fine-grained details and "ill-posed" regions. These include occlusions, where pixels are visible to only one camera; textureless regions, which lack matching features; and repetitive patterns that lead to ambiguous correspondences.

Existing end-to-end models frequently produce blurry results or lose sharp object boundaries because they often operate at reduced resolutions or attempt to solve the complex disparity mapping in a single pass. Cascade Residual Learning (CRL) addresses these limitations by decomposing the problem into an initial estimation and a subsequent refinement stage.

The CRL Architecture

The CRL framework consists of two primary sub-networks designed to rectify errors iteratively.

The Cascade Residual Learning (CRL) architecture, featuring DispFulNet for initial disparity and DispResNet for residual refinement.

Stage 1: DispFulNet

The first stage utilizes DispFulNet, an evolution of the DispNet architecture. Unlike its predecessor, which typically outputs disparity at half-resolution, DispFulNet incorporates additional up-convolution modules to generate a full-resolution initial disparity map. This provides a high-fidelity starting point for the refinement process.

Stage 2: DispResNet and Residual Learning

Following the initial estimation, a warping operation synthesizes a version of the left image from the right image using the predicted disparity. The difference between this synthesized image and the actual left image forms an explicit error map, highlighting exactly where the first stage failed.

The second stage, DispResNet, takes the original images, the initial disparity, and the error map as input. Rather than re-predicting the entire disparity map, DispResNet learns multiscale residuals—the corrections needed to reach the ground truth. The final output is the sum of the initial guess and these learned residuals. This approach is mathematically more efficient, as the network only needs to learn a "correction signal."

Experimental Results

The model was rigorously evaluated on three major benchmarks: FlyingThings3D (synthetic pre-training), Middlebury 2014 (high-resolution indoor), and KITTI 2015 (real-world driving).

Qualitative comparison between first-stage and second-stage outputs, highlighting the reduction in error maps.

Performance and Accuracy

At the time of its introduction, CRL ranked first on the KITTI 2015 stereo benchmark. It achieved a 3-pixel error (3PE) of 2.67%, significantly outperforming contemporary models like GC-NET (2.87%) and DRR (3.16%). Qualitatively, the refinement stage is particularly effective at recovering thin structures, such as poles and branches, which are often lost in single-stage architectures.

Efficiency

CRL processes a KITTI stereo pair in approximately 0.47 seconds on an Nvidia GTX 1080. While not yet real-time, this speed is considerably faster than many high-accuracy competitors that require several seconds or minutes per frame, making it a more practical choice for near-real-time applications.

Conclusion

Cascade Residual Learning demonstrates that explicit residual supervision at multiple scales is a powerful mechanism for stereo matching. By maintaining full resolution throughout the pipeline and focusing on error correction rather than redundant estimation, CRL sets a new standard for accuracy in challenging environments. Future work may involve incorporating left-right consistency checks to further improve robustness against occlusions and optimizing the architecture to reduce memory overhead.

Summary updated

Abstract

Leveraging on the recent developments in convolutional neural networks (CNNs), matching dense correspondence from a stereo pair has been cast as a learning problem, with performance exceeding traditional approaches. However, it remains challenging to generate high-quality disparities for the inherently ill-posed regions. To tackle this problem, we propose a novel cascade CNN architecture composing of two stages. The first stage advances the recently proposed DispNet by equipping it with extra up-convolution modules, leading to disparity images with more details. The second stage explicitly rectifies the disparity initialized by the first stage; it couples with the first-stage and generates residual signals across multiple scales. The summation of the outputs from the two stages gives the final disparity. As opposed to directly learning the disparity at the second stage, we show that residual learning provides more effective refinement. Moreover, it also benefits the training of the overall cascade network. Experimentation shows that our cascade residual learning scheme provides state-of-the-art performance for matching stereo correspondence. By the time of the submission of this paper, our method ranks first in the KITTI 2015 stereo benchmark, surpassing the prior works by a noteworthy margin.
BibTeX
@inproceedings{pang2017cascade,
  author = {Jiahao Pang and Wenxiu Sun and Jimmy S. J. Ren and Chengxi Yang and Qiong Yan},
  title = {{Cascade Residual Learning: A Two-Stage Convolutional Neural Network for Stereo Matching}},
  booktitle = {Workshops @ ICCV},
  year = {2017},
  doi = {10.1109/iccvw.2017.108},
}
Figure