Cross-Modal Analysis of Human Detection for Robotics: An Industrial Case Study

Timm Linder, Narunas Vaskevicius, Robert Schirmer, Kai O. Arras

Abstract

This study presents a comprehensive cross-modal comparison of human detection algorithms using 2D laser, 3D lidar, and RGB-D sensors in industrial environments. Addressing the scarcity of annotated 3D data, the authors propose a multi-sensor transfer learning strategy. By utilizing an RGB-D detector trained on synthetic data to generate labels for unlabeled 3D lidar data, the research demonstrates a significant performance boost in target-domain lidar detection without manual annotation.

Introduction

Reliable human detection is a cornerstone of safe and efficient mobile robotics, particularly in industrial settings like warehouses. While autonomous driving research has advanced 3D lidar-based detection, these models often struggle when transitioned to indoor environments due to domain gaps in sensor placement, clutter, and resolution. Furthermore, the high cost of manual 3D annotation for specific industrial sites remains a major barrier. This paper evaluates various sensor-algorithm combinations and introduces a method to adapt high-performance models to new domains using cross-modal supervision.

Methodology

The evaluation spans three primary modalities: 2D laser, 3D lidar, and RGB-D. While classical methods and modern deep learning are tested for 2D and 3D sensors, the core innovation lies in the Multi-Sensor Transfer Learning approach.

Multi-Sensor Transfer Learning

To bridge the domain gap for 3D lidar detectors without manual labeling, the authors employ a "teacher-student" framework. An RGB-D YOLO model, trained exclusively on synthetic data, acts as the teacher. This model predicts oriented 3D bounding boxes from RGB-D pairs. When deployed on unlabeled real-world data from the target environment, it generates "weak" labels that are then used to fine-tune a 3D lidar detector (specifically SECOND with Dynamic Voxelization).

The multi-sensor transfer learning pipeline uses synthetic-trained RGB-D models to supervise LiDAR detection in unlabeled target domains.

Experimental Setup

The study utilizes a custom industrial dataset recorded in a food factory warehouse. The mobile platform is equipped with a diverse sensor suite to capture the necessary modalities for comparison.

The mobile robot platform equipped with a Kinect v2 (RGB-D), VLP-16 (3D LiDAR), and S300 (2D safety laser) used for data collection in the industrial warehouse.

Datasets

  • Evaluation: Two challenging sequences from a warehouse: NCFM Dynamic (high density) and NCFM Storage Room (narrow aisles).
  • Training: KITTI for baseline lidar models, MS COCO combined with synthetic 3D data for the RGB-D teacher, and 27,000 unlabeled frames from the target warehouse for transfer learning.

Key Results and Insights

  • Performance: The RGB-D YOLO model was the top performer, achieving a 93.5% Peak-F1 score.
  • Transfer Learning Efficacy: Fine-tuning the 3D lidar model using the RGB-D teacher provided a substantial boost, increasing Average Precision (AP) by up to +11.5% in narrow storage environments compared to models trained only on the outdoor KITTI dataset.
  • Domain Gap: Models optimized for autonomous driving (KITTI) frequently fail indoors where humans are in close proximity to walls or equipment, highlighting the need for domain-specific adaptation.
  • Efficiency vs. Accuracy: While 3D methods offer better spatial awareness, 2D laser methods (like DR-SPAAM) remain highly efficient, running at ~48 Hz on a GPU, making them suitable for resource-constrained platforms.

Conclusion

This research underscores the power of synthetic data and cross-modal transfer learning in overcoming data scarcity for industrial robotics. While RGB-D sensors provide superior accuracy, their limited field of view remains a challenge. The proposed transfer learning method allows robots to leverage the strengths of camera-based detection to improve the robustness of 360° lidar systems in complex, cluttered indoor environments.

Summary updated

Abstract

Advances in sensing and learning algorithms have led to increasingly mature solutions for human detection by robots, particularly in selected use-cases such as pedestrian detection for self-driving cars or close-range person detection in consumer settings. Despite this progress, the simple question which sensor-algorithm combination is best suited for a person detection task at handƒ remains hard to answer. In this paper, we tackle this issue by conducting a systematic cross-modal analysis of sensor-algorithm combinations typically used in robotics. We compare the performance of state-of-the-art person detectors for 2D range data, 3D lidar, and RGB-D data as well as selected combinations thereof in a challenging industrial use-case.We further address the related problems of data scarcity in the industrial target domain, and that recent research on human detection in 3D point clouds has mostly focused on autonomous driving scenarios. To leverage these methodological advances for robotics applications, we utilize a simple, yet effective multi-sensor transfer learning strategy by extending a strong image-based RGB-D detector to provide cross-modal supervision for lidar detectors in the form of weak 3D bounding box labels.Our results show a large variance among the different approaches in terms of detection performance, generalization, frame rates and computational requirements. As our use-case contains difficulties representative for a wide range of service robot applications, we believe that these results point to relevant open challenges for further research and provide valuable support to practitioners for the design of their robot system.
BibTeX
@inproceedings{linder2021cross,
  author = {Timm Linder and Narunas Vaskevicius and Robert Schirmer and Kai O. Arras},
  title = {{Cross-Modal Analysis of Human Detection for Robotics: An Industrial Case Study}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=FpC2QSjRAH},
}
Figure