Abstract
3D object detection using LiDAR point clouds is a fundamental task in the fields of computer vision, robotics, and autonomous driving. However, existing 3D detectors heavily rely on annotated datasets, which are both time-consuming and prone to errors during the process of labeling 3D bounding boxes. In this paper, we propose a Scene Completion Pre-training (SCP) method to enhance the performance of 3D object detectors with less labeled data. SCP offers three key advantages: (1) Improved initialization of the point cloud model. By completing the scene point clouds, SCP effectively captures the spatial and semantic relationships among objects within urban environments. (2) Elimination of the need for additional datasets. SCP serves as a valuable auxiliary network that does not impose any additional efforts or data requirements on the 3D detectors. (3) Reduction of the amount of labeled data for detection. With the help of SCP, the existing state-of-the-art 3D detectors can achieve comparable performance while only relying on 20% labeled data.
Introduction
3D object detection serves as the "eyes" for autonomous systems, enabling them to identify vehicles, pedestrians, and obstacles within LiDAR point clouds. Despite its importance, training robust models is hindered by the high cost and error-prone nature of 3D bounding box annotation. Furthermore, LiDAR data is inherently sparse and subject to occlusions, making it difficult for models to learn complete object geometries.
While previous pre-training approaches often focus on individual object completion, they frequently overlook the broader environmental context. This paper introduces Scene Completion Pre-training (SCP), a method that leverages the spatial and semantic relationships within entire urban scenes to improve detection performance with significantly less labeled data.
Proposed Method: Scene Completion Pre-training
The SCP framework consists of a two-stage process designed to provide a superior initialization for 3D detectors.
Stage 1: Scene Completion Pre-training
In the first stage, the model is tasked with predicting a complete 3D scene from a partial, raw LiDAR point cloud.
- Voxelization: Raw points are converted into a structured 3D grid (voxels) to facilitate processing.
- Encoder: A Transformer-based backbone, VoTr (Voxel Transformer), is employed to extract features. The Transformer architecture is particularly effective at capturing long-range dependencies across the scene.
- Decoder: To reconstruct the scene, an Anisotropic Convolution (AIC) module is used. AIC dynamically adapts its receptive field, allowing it to handle the varying shapes and densities characteristic of 3D environments.
Stage 2: 3D Object Detection
The weights learned by the encoder during the scene completion task are used to initialize a 3D detector, such as VoTr-SSD or VoTr-TSD. The model is then fine-tuned using a limited set of labeled detection data. This "warm-up" phase allows the model to develop a "spatial common sense" regarding object placement and environmental structure before tackling the specific task of bounding box regression.
Experimental Results
The authors evaluated SCP using the SemanticKITTI dataset for pre-training and the KITTI 3D Object Detection Benchmark for downstream evaluation.
Label Efficiency and Performance
A key finding of this research is the "20% Rule." Models pre-trained with SCP and fine-tuned on only 20% of the KITTI labels achieved performance comparable to or better than baseline models trained on 100% of the labels. For instance, in the KITTI "Easy" car category, the VoTr-TSD-SCP model (using 20% data) achieved an Average Precision (AP) of 89.10%, surpassing the baseline VoTr-TSD trained on the same amount of data (89.04%).
Correlation Between Completion and Detection
The study revealed a direct correlation between the quality of scene completion and subsequent detection accuracy. Improvements in completion quality (measured by Intersection over Union) translated directly into higher downstream detection scores, confirming that better spatial understanding leads to better object localization.
Conclusion and Future Work
SCP demonstrates that scene completion is a powerful pretext task for 3D object detection, significantly reducing the reliance on expensive manual annotations. While current evaluations focus on the "Car" category, future research could extend this to more challenging classes like pedestrians and cyclists. Additionally, SCP could serve as a foundational pre-training step for other 3D tasks, including semantic segmentation and motion forecasting. Future efforts may also focus on optimizing the computational efficiency of the pre-training phase.
Summary updated