TL;DR
This paper introduces SECOND, a novel 3D object detection network for autonomous driving that significantly speeds up inference by employing sparse convolutional networks. It achieves state-of-the-art results on the KITTI dataset by introducing a new angle loss regression and a data augmentation strategy that leverages point cloud properties.
What Problem Does This Paper Address?
Detecting objects in 3D space is crucial for autonomous driving and robotics, with LiDAR data offering rich depth information. While 3D convolutional neural networks (CNNs) have shown promise, existing voxel-based methods suffer from slow inference speeds and suboptimal orientation estimation. This limits their real-time applicability. The paper aims to bridge this gap by developing a faster and more accurate 3D object detector that effectively utilizes LiDAR data.
Methods
The SECOND detector is a single-stage, end-to-end network with three main components:
- Voxelwise Feature Extractor: Raw point clouds are first voxelized. A Voxel Feature Encoding (VFE) layer then extracts features from points within each voxel, aggregating them to represent the voxel's local geometry.
- Sparse Convolutional Middle Layer: This is the core innovation. Instead of dense 3D convolutions, SECOND uses spatially sparse convolutions. This significantly reduces computation by only processing active voxels. The paper also introduces a GPU-based rule generation algorithm to accelerate sparse convolution. This layer learns features along the z-axis and then converts the sparse 3D data into a 2D Bird's Eye View (BEV) representation.
- Region Proposal Network (RPN): A modified SSD-like architecture takes the BEV feature maps and generates 3D bounding box proposals, predicting class, regression offsets, and object direction.
Key novelties include:
- Sparse Convolution: Dramatically speeds up training and inference compared to dense 3D convolutions.
- Angle Loss Regression: A new loss function that addresses the adversarial problem of predicting angles close to 0 or $\pi$, improving orientation estimation.
- Data Augmentation: A technique that samples ground truths and their associated point clouds from a database to create richer training data, improving convergence speed and performance.
Datasets
The paper uses the KITTI dataset for both training and evaluation. Specifically, it evaluates on the KITTI benchmarks for 3D object detection and BEV object detection, covering Car, Pedestrian, and Cyclist classes across easy, moderate, and hard difficulty levels.
Key Insights and Ideas
- Sparsity is Key for Speed: The paper demonstrates that exploiting the inherent sparsity of LiDAR point clouds through sparse convolutions is a highly effective strategy for achieving real-time performance in 3D object detection.
- Angle Regression is Tricky: The problem of predicting object orientation is non-trivial, especially with cyclical angles (0 and $\pi$ being equivalent). The proposed angle loss directly tackles this, leading to better localization.
- Data Augmentation Matters: The scarcity of ground truth data in real-world datasets is a significant bottleneck. The proposed sampling method effectively synthesizes training data, boosting model performance and reducing training time.
- LiDAR-Only is Viable: The work shows that high-performance 3D object detection can be achieved using LiDAR data alone, without relying on image fusion, which simplifies the system and potentially improves robustness.
Key Results
- Speed: The proposed sparse convolution achieves a 4x speed enhancement during training and a 3x improvement in inference speed compared to dense convolution on the KITTI dataset. A smaller model achieves a runtime of approximately 0.025s on a GTX 1080 Ti GPU.
- Accuracy: SECOND achieves state-of-the-art results on the KITTI 3D object detection benchmarks for the Car class, outperforming previous LiDAR-only methods. For pedestrian and cyclist detection, results are comparable to state-of-the-art.
- Performance: The large model runs at 20 fps, and the smaller model runs at 40 fps.
- Validation: On the KITTI validation set, SECOND (large) achieves 87.43% AP for easy car detection, and SECOND (small) achieves 85.50% AP for easy car detection, outperforming other methods.
Limitations and Future Work
- Pedestrian and Cyclist Performance: The paper notes that SECOND shows lower performance for pedestrian and cyclist detection, and for BEV detection, compared to car detection. This is attributed to the higher instance densities and smaller volumes of these objects, making them harder to distinguish from noise and limiting the power of CNNs.
- Future Work: The authors suggest several directions:
- Investigating joint camera- and LiDAR-based detection, potentially by fusing image features with LiDAR voxel features to enhance detection performance.
- Exploring weakly supervised training to leverage the large unlabeled portion of the KITTI dataset.
Summary updated