3D LiDAR and Stereo Fusion using Stereo Matching Network with Conditional Cost Volume Normalization

Tsun-Hsuan Wang, Hou-Ning Hu, Chieh Hubert Lin, Yi-Hsuan Tsai, Wei-Chen Chiu, Min SunIROS 2019 · OpenReview

TL;DR

This paper proposes a novel method for fusing sparse 3D LiDAR data with dense stereo camera information to improve depth estimation. By integrating LiDAR data directly into the stereo matching network's cost computation and cost volume regularization stages, the authors achieve state-of-the-art performance on challenging datasets, demonstrating a more effective fusion strategy than simply combining depth outputs.

What Problem Does This Paper Address?

Accurate 3D perception is crucial for applications like autonomous driving and robotics. While active sensors like LiDAR provide sparse but accurate depth information, and passive stereo cameras offer denser but less reliable depth maps (especially in challenging visual conditions), neither sensor is perfect on its own. Existing methods often fuse depth estimates at a late stage, failing to fully leverage the complementary strengths of these sensors. This paper addresses the challenge of effectively integrating sparse LiDAR data earlier into the stereo matching pipeline to improve the accuracy and robustness of dense depth estimation, particularly by resolving ambiguities inherent in stereo matching.

Methods

The core of the proposed method lies in two novel techniques integrated into a deep learning-based stereo matching network:

  • Input Fusion: This technique enriches the initial feature extraction stage by concatenating the stereo images with reprojected sparse LiDAR depth maps. This allows the network to jointly reason about appearance and geometric information from both modalities from the outset.
  • Conditional Cost Volume Normalization (CCVNorm): This is the key innovation. Instead of standard batch normalization, CCVNorm uses LiDAR information to dynamically modulate the cost volume features during the cost regularization stage. This helps to reduce the search space for matching and resolve ambiguities. The authors propose two variants:
  • Categorical CCVNorm: Uses a lookup table to map discretized LiDAR depth values to normalization parameters.
  • Continuous CCVNorm: Employs a CNN to learn the mapping from LiDAR data to normalization parameters.
  • Hierarchical CCVNorm: A more efficient extension that decomposes the modulation into hierarchical steps, significantly reducing parameter count while maintaining performance. This variant is highlighted for its practical benefits.

Datasets

The proposed method is evaluated on two standard datasets:

  • KITTI Stereo 2015 Dataset: Used for evaluating disparity estimation.
  • KITTI Depth Completion Dataset: Used for evaluating depth estimation, particularly for semi-dense ground truth.

The authors note that for the KITTI Depth Completion dataset, they train on a subset of non-overlapping scenes and use a split validation set for testing.

Key Insights and Ideas

  • Early Fusion is Key: Simply fusing depth outputs from LiDAR and stereo is suboptimal. Integrating LiDAR information earlier in the stereo matching pipeline (specifically in cost computation and cost volume regularization) unlocks greater potential.
  • LiDAR as an Ambiguity Resolver: Sparse LiDAR points are invaluable for disambiguating pixel correspondences that are difficult for stereo matching alone, especially in areas with repetitive textures or illumination changes.
  • Conditional Normalization for Targeted Regularization: CCVNorm's ability to adaptively modulate cost volume features based on LiDAR data is crucial for guiding the stereo matching process more effectively than generic regularization.
  • Hierarchical CCVNorm Balances Performance and Efficiency: The hierarchical extension of CCVNorm offers a compelling trade-off, achieving performance comparable to its more complex counterparts with significantly reduced computational overhead and model size.
  • Robustness to Sparsity: The CCVNorm approach demonstrates superior robustness to varying LiDAR data densities compared to other fusion methods, making it more practical for real-world scenarios where LiDAR data might be incomplete.

Key Results

  • The proposed method achieves state-of-the-art performance on both the KITTI Stereo 2015 and KITTI Depth Completion datasets, outperforming various baselines, including those that also incorporate LiDAR data.
  • On the KITTI Stereo dataset, the full model (Input Fusion + Hierarchical CCVNorm) achieves the best accuracy among all baselines.
  • For the KITTI Depth Completion dataset, the method shows significant improvements, particularly on inverse depth metrics (iRMSE, iMAE), which are critical for robotic applications.
  • The hierarchical CCVNorm variant provides a substantial reduction in computational time and model parameters compared to the original CCVNorm, with only a marginal increase in computation time (0.049 seconds) over the baseline GC-Net.
  • Qualitative results show the method's ability to capture fine details in complex structures, such as bicycles in low illumination, where other methods fail.

Limitations and Future Work

  • Discretization Error in Categorical CCVNorm: The paper notes that the categorical CCVNorm might suffer from quantization errors due to discretizing LiDAR depth, potentially impacting 1-pixel error metrics.
  • Void Information Handling: While addressed by CCVNorm, handling void information in sparse LiDAR data remains a challenge that the proposed method tackles.
  • Computational Overhead: While significantly reduced by the hierarchical extension, the proposed methods still introduce some computational overhead compared to a pure stereo matching network.
  • Future Work: The paper suggests that the proposed techniques could be further explored for other sensor fusion tasks and that optimizing stereo matching networks for even greater efficiency is an ongoing area of research. The authors also highlight the potential for their method to be applied in real-time scenarios.

Summary updated

Abstract

The complementary characteristics of active and passive depth sensing techniques motivate the fusion of the LiDAR sensor and stereo camera for improved depth perception. Instead of directly fusing estimated depths across LiDAR and stereo modalities, we take advantages of the stereo matching network with two enhanced techniques: Input Fusion and Conditional Cost Volume Normalization (CCVNorm) on the LiDAR information. The proposed framework is generic and closely integrated with the cost volume component that is commonly utilized in stereo matching neural networks. We experimentally verify the efficacy and robustness of our method on the KITTI Stereo and Depth Completion datasets, obtaining favorable performance against various fusion strategies. Moreover, we demonstrate that, with a hierarchical extension of CCVNorm, the proposed method brings only slight overhead to the stereo matching network in terms of computation time and model size.
BibTeX
@inproceedings{wang2019d,
  author = {Tsun-Hsuan Wang and Hou-Ning Hu and Chieh Hubert Lin and Yi-Hsuan Tsai and Wei-Chen Chiu and Min Sun},
  title = {{3D LiDAR and Stereo Fusion using Stereo Matching Network with Conditional Cost Volume Normalization}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=7488nAToV7},
}
Figure