Abstract
Stereo matching remains a fundamental challenge in computer vision, particularly in ill-posed regions such as occlusions and textureless surfaces. This paper introduces Cascade Residual Learning (CRL), a novel two-stage framework that iteratively refines disparity maps. By employing a full-resolution initial estimation followed by multiscale residual learning, CRL achieves state-of-the-art accuracy on major benchmarks while maintaining competitive computational efficiency.
Introduction
Calculating depth from stereo image pairs is essential for applications ranging from autonomous driving to robotics. While deep learning has significantly advanced the field, standard Convolutional Neural Networks (CNNs) often struggle with fine-grained details and "ill-posed" regions. These include occlusions, where pixels are visible to only one camera; textureless regions, which lack matching features; and repetitive patterns that lead to ambiguous correspondences.
Existing end-to-end models frequently produce blurry results or lose sharp object boundaries because they often operate at reduced resolutions or attempt to solve the complex disparity mapping in a single pass. Cascade Residual Learning (CRL) addresses these limitations by decomposing the problem into an initial estimation and a subsequent refinement stage.
The CRL Architecture
The CRL framework consists of two primary sub-networks designed to rectify errors iteratively.
Stage 1: DispFulNet
The first stage utilizes DispFulNet, an evolution of the DispNet architecture. Unlike its predecessor, which typically outputs disparity at half-resolution, DispFulNet incorporates additional up-convolution modules to generate a full-resolution initial disparity map. This provides a high-fidelity starting point for the refinement process.
Stage 2: DispResNet and Residual Learning
Following the initial estimation, a warping operation synthesizes a version of the left image from the right image using the predicted disparity. The difference between this synthesized image and the actual left image forms an explicit error map, highlighting exactly where the first stage failed.
The second stage, DispResNet, takes the original images, the initial disparity, and the error map as input. Rather than re-predicting the entire disparity map, DispResNet learns multiscale residuals—the corrections needed to reach the ground truth. The final output is the sum of the initial guess and these learned residuals. This approach is mathematically more efficient, as the network only needs to learn a "correction signal."
Experimental Results
The model was rigorously evaluated on three major benchmarks: FlyingThings3D (synthetic pre-training), Middlebury 2014 (high-resolution indoor), and KITTI 2015 (real-world driving).
Performance and Accuracy
At the time of its introduction, CRL ranked first on the KITTI 2015 stereo benchmark. It achieved a 3-pixel error (3PE) of 2.67%, significantly outperforming contemporary models like GC-NET (2.87%) and DRR (3.16%). Qualitatively, the refinement stage is particularly effective at recovering thin structures, such as poles and branches, which are often lost in single-stage architectures.
Efficiency
CRL processes a KITTI stereo pair in approximately 0.47 seconds on an Nvidia GTX 1080. While not yet real-time, this speed is considerably faster than many high-accuracy competitors that require several seconds or minutes per frame, making it a more practical choice for near-real-time applications.
Conclusion
Cascade Residual Learning demonstrates that explicit residual supervision at multiple scales is a powerful mechanism for stereo matching. By maintaining full resolution throughout the pipeline and focusing on error correction rather than redundant estimation, CRL sets a new standard for accuracy in challenging environments. Future work may involve incorporating left-right consistency checks to further improve robustness against occlusions and optimizing the architecture to reduce memory overhead.
Summary updated