Multi-Scale Cost Volumes Cascade Network for Stereo Matching

Xiaogang Jia, Wei Chen, Chen Li, Zhengfa Liang, Mingfei Wu, Yusong Tan, Libo HuangICRA 2021 · DOI

Introduction

Stereo matching—the process of calculating depth by comparing two offset images—is a cornerstone of robot navigation and 3D reconstruction. However, current research faces a significant trade-off: traditional methods like Semi-Global Matching (SGM) are computationally efficient but struggle with lighting changes and textureless regions, while state-of-the-art Convolutional Neural Networks (CNNs) achieve high accuracy at the cost of massive memory consumption and high latency. These CNNs often rely on 4D cost volumes and 3D convolutions, which are frequently too slow for real-time robotics.

To address this, the authors propose MSCVNet (Multi-Scale Cost Volumes Cascade Network), a hybrid architecture that maintains the speed of 2D convolutions while recovering the accuracy typically lost when compressing feature information. MSCVNet achieves real-time performance (41ms) on standard hardware, offering a viable alternative for high-speed navigation.

Architecture and Methodology

MSCVNet's architecture is built around three primary innovations designed to balance efficiency and precision.

Hybrid Multi-Scale Cost Volumes

Instead of a single, computationally expensive 4D volume, the network generates multiple 3D cost volumes through two parallel branches:

  1. Traditional Branch: Utilizes a 5x5 Census transform for the Y channel and Absolute Difference (AD) for U/V channels. This provides stable, hand-crafted geometric cues.
  2. Neural Branch: Employs a U-Net feature extractor followed by a 1D Correlation layer to capture learned semantic features.

Combining these branches prevents feature redundancy and ensures the network does not have to "re-learn" basic geometric principles from scratch.

Overview of the MSCVNet architecture showing the dual-branch cost volume generation and the cascade refinement stages.

Guided Cascade Hourglass Network

Cost aggregation—the process of refining initial depth estimates—is performed using a 2D cascade hourglass structure. In this setup, the traditional cost volume acts as a guide, providing multi-scale features that assist the neural branch in refining the disparity map across successive stages. This multi-scale approach allows the network to capture both fine details and global context without the overhead of 3D convolutions.

Detailed view of the multi-scale cascade mechanism, illustrating how cost volumes are processed through ResNet blocks and upsampled across different resolutions.

Discontinuous Area Loss

To combat "edge fattening" or blurring at object boundaries, the authors introduced a novel algorithm to identify "discontinuous areas" (e.g., the edges of vehicles or poles). By checking if the warped disparity map maintains mathematical monotonicity, the network can automatically detect occlusions and boundaries. A specific loss function is then applied to these regions to sharpen the final output.

Experimental Results

The model was evaluated on the Scene Flow synthetic dataset and the real-world KITTI 2012 and 2015 benchmarks.

  • Efficiency: MSCVNet processes frames in 41ms on an NVIDIA 1080Ti. This makes it approximately 24x faster than CSPN and 44x faster than GANet.
  • Accuracy: On the KITTI 2015 benchmark, it achieved a D1-all error of 2.82%. While slightly higher than the most complex 3D-convolution models (which often hover around 1.8%), it represents a significant improvement over other real-time 2D networks like DispNetC.
  • Ablation Studies: Tests confirmed that the hybrid approach (Traditional + Neural) outperformed either method used in isolation, achieving a PC3 of 98.54%.

Conclusion and Future Work

MSCVNet demonstrates that integrating traditional computer vision techniques with optimized 2D CNN architectures can bridge the gap between real-time performance and high-accuracy depth estimation. By utilizing monotonicity as a signal for edge detection and leveraging multi-scale 3D volumes, the network provides a robust solution for robotics. Future work will focus on extending this architecture to Multi-View Stereo and integrating depth completion for LiDAR-enhanced navigation systems.

Summary updated

BibTeX
@inproceedings{jia2021multi,
  author = {Xiaogang Jia and Wei Chen and Chen Li and Zhengfa Liang and Mingfei Wu and Yusong Tan and Libo Huang},
  title = {{Multi-Scale Cost Volumes Cascade Network for Stereo Matching}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2021},
  doi = {10.1109/icra48506.2021.9560864},
}
Figure