Cascaded Multi-scale and Multi-dimension Convolutional Neural Network for Stereo Matching

Haihua Lu, Hai Xu, Li Zhang, Yanbo Ma, Yong Zhao

Abstract

Stereo matching is a fundamental task in computer vision, essential for depth perception in autonomous systems. This paper presents a novel cascaded CNN architecture that addresses the trade-off between global context and fine-grained spatial detail. The proposed model utilizes parallel feature extraction "stems" to capture multi-scale information and a hybrid 3D/2D convolutional aggregation sub-network to refine matching costs. Evaluated on the KITTI benchmarks, the architecture achieves high-accuracy disparity maps with minimal post-processing, demonstrating the effectiveness of combining geometric consistency with semantic understanding.

Introduction

Stereo matching—the process of calculating depth by comparing two slightly offset images—is a cornerstone of computer vision for autonomous vehicles and robotics. Despite significant improvements from deep learning, two persistent challenges remain in the field:

  1. The Receptive Field Trade-off: To accurately interpret "ill-posed" regions, such as featureless walls or repetitive patterns, a network requires a large receptive field for context. However, expanding this field often results in a loss of fine-grained spatial detail.
  2. Inefficient Cost Aggregation: The "cost volume" generated after matching pixels is frequently noisy. Early CNN approaches either bypassed refinement or relied on computationally expensive 3D convolutions that struggled to capture high-level semantic meaning.

Proposed Architecture

The authors introduce a cascaded architecture divided into two primary sub-networks: multi-scale matching cost computation and multi-dimension cost aggregation.

Overview of the proposed cascaded CNN architecture featuring parallel stems (A and B) and a hybrid aggregation sub-network.

Multi-scale Matching Cost Computation

Instead of relying on a single feature extractor, the model employs two parallel "stems" to process input images:

  • Stem A: A shallower encoder-decoder structure optimized to preserve fine spatial details.
  • Stem B: A deeper structure with additional pooling layers, providing a significantly wider receptive field to capture global context.

Feature fusion is achieved by concatenating deconvolution features with corresponding encoder maps via skip connections. This ensures that both high-level coarse information and low-level fine details are preserved throughout the matching process.

Multi-dimension Cost Aggregation

Once the initial cost volume is constructed, it is refined through a novel two-stage "multi-dimension" approach:

  • 3D Convolutional Stage: This stage operates on height, width, and disparity dimensions simultaneously to incorporate geometric and spatial consistency.
  • 2D Convolutional Stage: The cost volume is transposed and processed by a 2D CNN. This stage is specifically designed to capture "semantic" information, helping the network resolve ambiguous matches by understanding the broader object context.
Detailed layer specifications for the 3D (Agg_3D) and 2D (Agg_2D) aggregation sub-networks.

Experimental Results

The model was evaluated on the industry-standard KITTI 2012 and KITTI 2015 benchmarks, which feature real-world driving scenes with LIDAR-based ground truth.

  • Accuracy: On the KITTI 2015 benchmark, the model achieved a 3.40% error rate (D1-all) on non-occluded pixels, outperforming established models like Content-CNN and DispNetC.
  • Ablation Study: The multi-dimension aggregation (3D + 2D) significantly reduced errors compared to using 3D convolutions alone, dropping the 3-pixel error from 3.95% to 3.01% on KITTI 2015.
  • Efficiency: The model maintains a runtime of approximately 1.2 seconds, offering a competitive balance between the speed of real-time models and the high accuracy of heavy-duty architectures like MC-CNN-acrt (67 seconds).

Key Insights and Discussion

  • Parallelism over Sequentiality: Running two scales in parallel allows the network to dynamically balance context and detail rather than forcing a single-scale compromise. The authors argue that this dual-path approach makes the best use of both local and global variants.
  • Hybrid Aggregation: Combining 3D convolutions (for geometry) with 2D convolutions (for semantics) provides a more holistic refinement of the cost volume than either method alone.
  • End-to-End Simplicity: The architecture's robustness allows for a simple "Winner-Take-All" (WTA) strategy to produce competitive results, eliminating the need for complex, multi-step traditional post-processing pipelines.

Limitations and Future Work

The model's primary limitation is its performance in occlusion regions—areas visible in only one image. Currently, the network relies on matching cost and aggregation without a dedicated occlusion-handling module. Future work aims to design a robust post-processing module specifically for these areas to further enhance safety for autonomous driving applications.

Summary updated

BibTeX
@inproceedings{lu2018cascaded,
  author = {Haihua Lu and Hai Xu and Li Zhang and Yanbo Ma and Yong Zhao},
  title = {{Cascaded Multi-scale and Multi-dimension Convolutional Neural Network for Stereo Matching}},
  booktitle = {VCIP},
  year = {2018},
  doi = {10.1109/vcip.2018.8698637},
}
Figure