Self-Supervised Person Detection in 2D Range Data using a Calibrated Camera

Dan Jia, Mats Steinweg, Alexander Hermans, Bastian Leibe

Abstract

Deep learning is the essential block of state-of-the-art person detectors in 2D range data. However, only a few annotated datasets are available for training and testing these deep networks, potentially limiting their performance when deployed in new environments or with different LiDAR models. We propose a method, which uses bounding boxes from an image-based detector (e.g. Faster R-CNN) on a calibrated camera to automatically generate training labels (called pseudo-labels) for 2D LiDAR-based person detectors. Through experiments on the JackRabbot dataset with two detector models, DROW3 and DR-SPAAM, we show that self-supervised detectors, trained or fine-tuned with pseudo-labels, outperform detectors trained only on a different dataset. Combined with robust training techniques, the self-supervised detectors reach a performance close to the ones trained using manual annotations of the target dataset. Our method is an effective way to improve person detectors during deployment without any additional labeling effort.

Introduction

While deep learning has revolutionized object detection, its success depends on massive amounts of annotated data. For 2D LiDAR—a staple sensor in mobile robotics due to its wide field of view and low cost—high-quality datasets are scarce and difficult for humans to label compared to images. Consequently, LiDAR detectors trained on one dataset often fail when deployed in a new environment or with a different sensor model. This "domain gap" is a major hurdle for reliable robotic navigation. This paper proposes a self-supervised method to bridge this gap by letting the robot "teach itself" using its onboard calibrated camera.

Methodology: The Pseudo-Labeling Pipeline

The core of the paper is an automated pipeline to generate training labels (pseudo-labels) for 2D LiDAR data without human intervention:

  1. Image-Based Detection: An off-the-shelf detector (Faster R-CNN) identifies people in RGB images.
  2. Frustum Projection: Using known extrinsic calibration, the 2D LiDAR points are projected into the image. Points falling within the bottom half of a detected person's bounding box are extracted (targeting the legs/lower body).
  3. Spatial Clustering: To isolate the person from the background, the system applies $k$-means clustering ($k=2$) in the range space, followed by a Mean Shift procedure to pinpoint the person's exact center.
  4. Robust Training: Because pseudo-labels are inherently noisy, the authors employ Partially Huberized Cross-Entropy Loss to clip gradients for outliers and Mixup Regularization to improve generalization.
Success and failure cases of the pseudo-labeling pipeline, showing RGB crops and projected LiDAR points.

Experimental Evaluation

The primary evaluation is conducted on the JackRabbot Dataset (JRDB), which features 360° LiDAR scans and calibrated RGB images from diverse indoor and outdoor environments. The authors tested two state-of-the-art detectors: DROW3 and DR-SPAAM.

Key Results and Insights

  • Pseudo-labels are "Good Enough": The generated pseudo-labels achieved over 90% True Positive and True Negative rates, providing a signal strong enough to train specialized LiDAR models.
  • Overcoming the Domain Gap: A model trained on pseudo-labels in the target environment outperformed a model pre-trained on a different, manually labeled dataset. This proves that local "noisy" data is often more valuable than distant "clean" data.
  • Closing the Performance Gap: When fine-tuning a pre-trained model with pseudo-labels and robust losses, the system reached 76.4% $AP_{0.3}$, which is remarkably close to the performance of a model trained on manual 3D annotations (76.2% $AP_{0.3}$).
Comparison of detector performance (AP) over training updates using pseudo-labels versus manual annotations.
  • Online Adaptation: The paper demonstrates that a robot can fine-tune its detector "on the fly." As shown in the training curves, performance can jump significantly in just a few hundred updates when adapting to a specific sequence.

Limitations and Future Work

The method relies on the overlap between the camera and LiDAR fields of view and is sensitive to extrinsic calibration accuracy. Furthermore, at long distances, LiDAR points become sparse, making spatial clustering less reliable. Future directions include exploring Curriculum Learning to handle performance fluctuations during online fine-tuning, potentially allowing for more stable learning as robots navigate new spaces.

Summary updated

Abstract

Deep learning is the essential building block of state-of-the-art person detectors in 2D range data. However, only a few annotated datasets are available for training and testing these deep networks, potentially limiting their performance when deployed in new environments or with different LiDAR models. We propose a method, which uses bounding boxes from an image-based detector (e.g. Faster R-CNN) on a calibrated camera to automatically generate training labels (called pseudo-labels) for 2D LiDAR-based person detectors. Through experiments on the JackRabbot dataset with two detector models, DROW3 and DR-SPAAM, we show that self-supervised detectors, trained or fine-tuned with pseudo-labels, outperform detectors trained only on a different dataset. Combined with robust training techniques, the self-supervised detectors reach a performance close to the ones trained using manual annotations of the target dataset. Our method is an effective way to improve person detectors during deployment without any additional labeling effort, and we release our source code to support relevant robotic applications.
BibTeX
@inproceedings{jia2021self,
  author = {Dan Jia and Mats Steinweg and Alexander Hermans and Bastian Leibe},
  title = {{Self-Supervised Person Detection in 2D Range Data using a Calibrated Camera}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2021},
  doi = {10.1109/icra48506.2021.9561699},
}
Figure