TL;DR
This paper introduces a robust approach for person segmentation and action recognition (walking, waving, sitting) using hemispheric field-of-view LiDAR sensors. By projecting multi-channel LiDAR data into 2D representations and leveraging a modified MaskDINO architecture, the researchers achieve high-precision perception that is both real-time capable and inherently privacy-preserving.
What Problem Does This Paper Address?
For service robots to operate safely in human environments, they must "see" people and understand their activities. While RGB cameras are the standard tool for this, they face two major hurdles: privacy and reliability. Cameras capture facial details that many users find intrusive, and they struggle in low-light or high-contrast conditions.
LiDAR is a promising alternative because it provides precise 3D measurements regardless of lighting and lacks the resolution to "leak" sensitive facial features. However, most LiDAR research focuses on autonomous driving (long-range, narrow vertical FoV). This paper tackles the challenges of hemispheric LiDARs (like the Ouster OSDome), which provide a 180° "bubble" of vision. These sensors have a non-uniform beam distribution—denser near the axis of rotation and sparser at the periphery—which can distort traditional computer vision models.
Methods
The authors propose a pipeline that treats LiDAR data as a multi-channel image rather than a raw point cloud.
- Representation: They project the 3D scan into a 512x64 2D matrix where the axes represent the rotation angle and beam number.
- Multi-Channel Input: The model uses four standard LiDAR channels: Range, Signal Intensity, Reflectivity, and Near-Infrared (NIR).
- Positional Encoding: To help the model understand the sensor's distorted geometry, they add three additional channels representing the Euclidean XYZ coordinates of each point, resulting in a 7-channel input.
- Architecture: They adapted MaskDINO, a state-of-the-art transformer-based framework for joint detection and segmentation, using a SwinL (Swin Transformer Large) backbone.
- Action Recognition: The model was fine-tuned to not only segment the person but also classify their state into one of three categories: walking, waving, or sitting.
Datasets
The researchers introduced a newly recorded and annotated dataset because existing LiDAR benchmarks (like SemanticKITTI) focus on outdoor automotive scenes.
- Hardware: Data was collected using an Ouster OSDome-64 mounted on a TIAGo++ omnidirectional mobile robot.
- Content: 442 scans featuring individuals performing specific actions in indoor environments.
- Availability: The authors have made the dataset, trained models, and code publicly available to the research community.
Key Insights and Ideas
- Privacy by Design: The authors argue that by using sensors that don't measure high-resolution imagery in the first place, it becomes impossible to leak private information through model weights or training data.
- The Power of Positional Encoding: Adding XYZ coordinates as input channels was crucial. It allowed the 2D CNN-based backbone to "understand" the 3D spatial relationships that are warped during the 2D projection of a hemispheric sensor.
- Channel Contribution: An ablation study revealed that Reflectivity and Range are the most critical channels for person detection. Interestingly, NIR and Signal intensity provided only minor contributions, and in some cases, removing them actually improved performance, suggesting that simpler input representations might be more efficient.
- Cross-Sensor Generalization: Although trained on a 64-beam sensor, the model performed well on a 128-beam sensor (OSDome-128) without retraining, proving the approach is robust to changes in sensor resolution.
Key Results
- Segmentation Accuracy: The model achieved an F1-score of 0.97 for person segmentation when the backbone was allowed to train (unfrozen).
- Action Classification: The system performed exceptionally well across all three actions, with F1-scores consistently above 0.90.
- Real-Time Performance: On a mobile-grade GPU (Nvidia RTX A4500), the system achieved an inference rate of 11 Hz. This is fast enough for real-time human-robot interaction and tracking.
Limitations and Future Work
The primary limitation is the dataset size; 442 scans is relatively small for deep learning, though the results suggest it was sufficient for these specific tasks. The authors also noted that their "one-layer convolutional encoder" might not be the most efficient way to merge the 7 input channels.
Future work will likely involve exploring more sophisticated encoders to better handle multi-channel data and expanding the library of recognized actions to include more complex human behaviors.
Summary updated