TL;DR
This paper introduces Correlate-and-Excite (CoEx), a novel real-time stereo matching network that significantly improves efficiency without sacrificing accuracy. CoEx achieves this through two key innovations: Guided Cost Volume Excitation (GCE) for efficient feature utilization and a top-k disparity regression method that enhances depth estimation accuracy.
What Problem Does This Paper Address?
Stereo matching, the task of estimating depth from a pair of images, is crucial for applications like robotics and autonomous driving. While deep learning has dramatically improved accuracy, many state-of-the-art methods are too slow for real-time use. Existing approaches to speed up stereo matching often compromise accuracy. Volumetric methods, which use 3D convolutions to aggregate cost volumes, are computationally expensive and memory-intensive. While some methods try to improve efficiency with spatially varying aggregation, they often add complexity and computational overhead. This paper aims to bridge the gap between high accuracy and real-time performance in stereo matching.
Methods
The CoEx network is built upon two main novel contributions:
- Guided Cost Volume Excitation (GCE): Instead of complex spatially varying aggregation, GCE uses image features to "excite" the cost volume channels. This is a lightweight operation that allows the 3D convolutions to better leverage geometric information from the cost volume by guiding which features are most relevant. It's implemented using 2D point-wise convolutions and a sigmoid function to generate weights that are broadcast across disparity channels.
- Top-k Disparity Regression: Traditional methods use soft-argmin to compute disparity from the cost volume's probability distribution. However, this can be problematic with multi-modal or flat distributions. CoEx proposes using only the top-k values from the disparity distribution to compute the final disparity estimate. This simple yet effective method can suppress erroneous values and lead to more accurate depth predictions, especially in ambiguous regions.
The overall architecture follows a GC-Net-like hourglass structure with 3D convolutions for cost aggregation, but with reduced channels and depth for efficiency. The feature extraction uses a shared MobileNetV2 backbone with a U-Net style upsampling module.
Datasets
The paper evaluates CoEx on the following datasets:
- SceneFlow: A large synthetic dataset for training and evaluation, used for most of the ablation studies.
- KITTI Stereo 2012 & 2015: Real-world datasets with LiDAR-based ground truth, used for benchmarking against state-of-the-art methods.
Key Insights and Ideas
- Simplicity is Key for Efficiency: The paper demonstrates that complex spatially varying aggregation isn't always necessary. A simple channel-wise excitation (GCE) guided by image features can achieve similar or better results with significantly less computational overhead.
- Top-k is More Robust than Soft-Argmin: The top-k disparity regression method proves to be more robust than the standard soft-argmin, especially in cases with ambiguous or multi-peaked disparity distributions, leading to improved accuracy.
- Correlation-Based Cost Volumes Can Be Competitive: By using GCE, the authors show that correlation-based cost volumes (which are more efficient than concatenation-based ones) can achieve comparable accuracy to concatenation-based methods, further boosting efficiency.
- Real-time Performance is Achievable Without Major Accuracy Sacrifices: CoEx successfully balances speed and accuracy, outperforming many speed-oriented methods and remaining competitive with state-of-the-art, more computationally intensive models.
Key Results
- CoEx achieves competitive accuracy on SceneFlow, KITTI 2012, and KITTI 2015 datasets.
- The model is real-time, achieving a runtime of 27 ms on an RTX 2080Ti GPU, which is significantly faster than many other accurate methods.
- Compared to StereoNet (a fast model), CoEx offers substantially better accuracy (0.411 EPE difference on SceneFlow and 2.7% on KITTI 2015).
- CoEx is 3.3x faster than AANet while achieving 0.18 EPE lower and better KITTI metrics.
- The GCE module, when integrated into a baseline PSMNet, brings its accuracy to a similar level as concatenation-based PSMNet, but with significantly reduced computation time.
- The top-k regression, even with a small k (e.g., k=2), shows improved accuracy over soft-argmin.
Limitations and Future Work
- Top-k Regression Computation Cost: While the paper computes top-k regression at a reduced resolution (1/4 of input image) to mitigate computational cost, the authors note that performing this at full resolution would be too computationally expensive. Future work could explore more efficient ways to implement top-k regression at higher resolutions.
- Training for Top-k: The paper highlights that models need to be trained with the top-k regression to benefit from it; simply applying it at test time to a model trained with soft-argmin did not yield improvements. This suggests that the network needs to learn to produce distributions that are well-suited for top-k selection.
- Further Exploration of GCE: While GCE is shown to be effective, further research could explore different ways to generate guidance signals or apply them to other network architectures.
Summary updated