YOLOStereo3D: A Step Back to 2D for Efficient Stereo 3D Detection

Lujia Wang, Yuxuan Liu, Lujia Wang 0001, Ming Liu

TL;DR

This paper introduces YOLOStereo3D, a novel framework for efficient stereo 3D object detection that re-frames the problem as an enhanced monocular detection task. By leveraging insights from real-time 2D detectors and introducing a lightweight stereo matching module, YOLOStereo3D achieves competitive performance with state-of-the-art methods while running significantly faster and without requiring LiDAR data.

What Problem Does This Paper Address?

Accurate 3D object detection is crucial for autonomous systems like self-driving cars and mobile robots. While stereo cameras offer a cost-effective alternative to LiDAR, existing stereo 3D detection methods often rely on dense depth reconstruction from disparity estimation. This approach is computationally expensive, hindering real-world deployment on resource-constrained platforms. The paper aims to bridge this gap by developing a faster and more efficient stereo 3D detection system that can rival the accuracy of more complex methods.

Methods

YOLOStereo3D takes a unique approach by building upon the success of real-time 2D object detectors. Instead of directly tackling the computationally intensive dense depth estimation, it treats stereo 3D detection as a monocular 3D detection problem enhanced with stereo features. Key methodological contributions include:

  • Lightweight Stereo Matching Module: A novel, efficient stereo matching module is introduced. It utilizes a pixel-wise correlation mechanism to construct a "thin" cost volume, significantly reducing computational cost compared to traditional concatenation-based methods.
  • Hierarchical Multi-scale Fusion: The framework employs a hierarchical structure to fuse stereo features extracted at different scales. This approach balances information preservation with computational efficiency, ensuring that both fine-grained local features and broader semantic context are utilized.
  • Densely Connected Ghost Module: To further enhance feature representation and rebalance channel counts during fusion, a densely connected ghost module is incorporated. This module efficiently generates redundant features, improving the network's ability to capture rich information.
  • Anchor Preprocessing and Statistical Priors: The method leverages insights from monocular 3D detection by defining 3D anchors and incorporating statistical priors derived from the training data. This includes predicting transformed depth outputs and using scene-specific knowledge to filter anchors, leading to improved regression accuracy.
  • Auxiliary Disparity Supervision: During training, an auxiliary disparity estimation branch is used with a sparse ground truth disparity. This supervision helps guide the network to produce better stereo features, preventing it from getting stuck in a monocular-only local minimum.

Datasets

The primary dataset used for training and evaluation is KITTI Object Detection Benchmark. The authors also mention using a split of the KITTI dataset (Chen's split) for hyperparameter tuning and ablation studies.

Key Insights and Ideas

  • "Step Back to 2D" Philosophy: The core insight is that treating stereo 3D detection as an enhanced monocular problem, rather than a direct 3D reconstruction problem, can lead to significant efficiency gains without sacrificing accuracy.
  • Efficiency through Pixel-wise Correlation: Replacing concatenation-based cost volumes with pixel-wise correlation is a critical step for achieving real-time performance in stereo matching.
  • Hierarchical Fusion for Balance: A multi-scale hierarchical fusion strategy effectively balances the need for detailed disparity information with the computational constraints of real-time processing.
  • Anchor Priors Matter: Incorporating statistical priors and intelligent anchor preprocessing significantly improves the robustness and accuracy of 3D object detection, even in a stereo setting.
  • Disparity Supervision as Regularization: Auxiliary disparity supervision acts as a crucial regularizer, guiding the network towards better stereo feature learning and preventing it from defaulting to a less effective monocular approach.

Key Results

  • Speed: YOLOStereo3D achieves an inference speed of over 10 frames per second on a single GPU, a significant improvement over existing methods that often take more than 0.3 seconds per frame.
  • Accuracy: The model demonstrates competitive performance on the KITTI 3D benchmark, comparable to state-of-the-art stereo 3D detection frameworks, without the use of LiDAR data. For instance, on the "Car" category, it achieves 65.68%/41.25%/30.42% for Easy/Moderate/Hard IoU thresholds.
  • Efficiency: The framework requires significantly less GPU memory (around 7 GB for a batch size of 4) and a shorter training time compared to other SOTA stereo detection algorithms.
  • Monocular Performance: The anchor preprocessing techniques also prove effective in monocular 3D detection, achieving state-of-the-art results with a running time of about 50 ms per frame.

Limitations and Future Work

  • Asymmetric Image Roles: The paper acknowledges that the current framework converts features from the right image to the left image, making the computational roles of the two images unequal. This can lead to significant information loss from the right image, potentially making the model sub-optimal when objects are occluded in the left image but visible in the right.
  • Future Work: While not explicitly detailed, the authors suggest that YOLOStereo3D lowers the bar for stereo detection research and can boost the deployment of stereo setups on self-driving cars and mobile robots. Future work could explore more symmetrical handling of stereo image information or further optimization of the fusion process.

Summary updated

Abstract

Object detection in 3D with stereo cameras is an important problem in computer vision, and is particularly crucial in low-cost autonomous mobile robots without LiDARs. Nowadays, most of the best-performing frameworks for stereo 3D object detection are based on dense depth reconstruction from disparity estimation, making them extremely computationally expensive. To enable real-world deployments of vision detection with binocular images, we take a step back to gain insights from 2D image-based detection frameworks and enhance them with stereo features. We incorporate knowledge and the inference structure from real-time one-stage 2D/3D object detector and introduce a light-weight stereo matching module. Our proposed framework, YOLOStereo3D, is trained on one single GPU and runs at more than ten fps. It demonstrates performance comparable to state-of-the-art stereo 3D detection frameworks without usage of LiDAR data. The code will be published in https://github.com/Owen-Liuyuxuan/visualDet3D.
BibTeX
@inproceedings{wang2021yolostereo,
  author = {Lujia Wang and Yuxuan Liu and Lujia Wang and Ming Liu},
  title = {{YOLOStereo3D: A Step Back to 2D for Efficient Stereo 3D Detection}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2021},
  doi = {10.1109/icra48506.2021.9561423},
}
Figure