TL;DR
This paper introduces MobileStereoNet, a novel family of lightweight deep networks for stereo matching designed for resource-constrained devices. By adapting MobileNet blocks for 3D convolutions and proposing a new interlacing cost volume, they achieve competitive accuracy with significantly reduced computational cost and memory footprint.
What Problem Does This Paper Address?
Stereo matching, a crucial technique for depth perception in computer vision, has seen significant accuracy improvements with deep learning. However, these state-of-the-art models often come with a prohibitive computational cost, making them unsuitable for deployment on embedded systems, mobile devices, or platforms with limited GPU resources. This leads to an "out-of-memory" (OOM) state or requires specialized hardware. The paper aims to bridge this gap by developing stereo matching networks that are both accurate and computationally efficient, enabling their use in real-world applications like robotics and autonomous driving.
Methods
The core of MobileStereoNet lies in two main innovations:
- Lightweight 3D Convolutions: The authors adapt the efficient MobileNet-V1 and MobileNet-V2 architectures, originally designed for 2D image tasks, to handle 3D convolutions. This involves extending their depthwise and pointwise operations to operate across the depth dimension of the cost volume. This adaptation significantly reduces the number of parameters and operations compared to standard 3D convolutions while maintaining performance.
- Interlacing Cost Volume Construction: To improve the accuracy of the 2D MobileStereoNet model and reduce its reliance on computationally expensive 3D convolutions, a novel "Interlacing Cost Volume" module is proposed. This module learns to aggregate features from left and right images by interlacing their unary features and then processing them with a series of 3D convolutions. This parameterized approach aims to achieve better similarity measurements between features across the disparity dimension compared to traditional methods like concatenation or simple correlation.
The paper also details how these components are integrated into two baseline architectures: a 2D baseline and a 3D baseline, which are then modified using the proposed lightweight blocks and cost volume.
Datasets
The paper utilizes the following datasets:
- SceneFlow "final pass" dataset: Used for pre-training the networks and for extensive experiments analyzing design choices. This dataset contains 35,454 training and 4,370 test samples with a resolution of 540x960.
- KITTI 2015 dataset: Used for fine-tuning and evaluating the models on real-world driving scenarios. This dataset has 200 training samples and is used with a 159/40 training/validation split.
Key Insights and Ideas
- MobileNet's 3D Potential: The research demonstrates that the efficiency principles behind MobileNet architectures can be effectively translated to the 3D domain for stereo matching, offering a significant reduction in computational load without sacrificing accuracy.
- Parameterized Cost Volume for 2D: The interlacing cost volume proves to be a critical component for the 2D MobileStereoNet, allowing it to achieve accuracy comparable to more complex 3D models by learning better feature aggregation.
- Trade-offs in Expansion Factor (t): For MobileNet-V2 blocks, the expansion factor (t) plays a crucial role. While increasing 't' can improve accuracy, it also increases computational cost. The paper identifies an optimal range for 't' to balance these factors.
- Feature Extraction as a Bottleneck: The feature extraction backbone is identified as a major contributor to the computational load in stereo matching networks, making it a prime candidate for optimization with lightweight blocks.
Key Results
- Computational Efficiency:
- The proposed 2D and 3D networks reduce computational expense by 27%/95% and 72%/38% fewer parameters/operations, respectively, compared to their counterparts.
- Specifically, 3D-MobileStereoNet uses 1.7x fewer parameters and 3.9x fewer GigaMACs than GwcNet-gc.
- 2D-MobileStereoNet achieves comparable results to PSMNet (a 3D model) with 2.3x/8x fewer parameters/operations.
- Accuracy:
- The proposed networks achieve competitive accuracy on the SceneFlow dataset, with the interlaced cost volume (specifically
Interlaced4) showing the best performance.
- The proposed networks achieve competitive accuracy on the SceneFlow dataset, with the interlaced cost volume (specifically
- On the KITTI 2015 benchmark, 3D-MobileStereoNet shows superior performance compared to GCNet and PSMNet.
- Model Size: The proposed MobileStereoNets yield significantly smaller models, with 3D-MobileStereoNet requiring only 7.99MB and 2D-MobileStereoNet 10.03MB, making them highly suitable for memory-constrained devices.
Limitations and Future Work
- Limitations:
- While the paper focuses on reducing complexity, the 2D baseline still requires a 3D cost volume, which is then processed by 2D convolutions. This is a compromise to enable the use of 2D convolutions in the encoder-decoder.
- The paper notes that replacing the channel reduction module and convolutions in the cost volume construction with lightweight blocks in the 2D baseline can deteriorate the model's learning capability.
- Future Work: The authors suggest that their findings on design choices and the proposed lightweight architectures are applicable to other 2D/3D stereo networks, paving the way for further research into efficient stereo matching. The success of MobileStereoNets makes them promising candidates for deployment on edge devices.
Summary updated