Person Segmentation and Action Classification for Multi-Channel Hemisphere Field of View LiDAR Sensors

Svetlana Seliunina, Artem Otelepko, Raphael Memmesheimer, Sven Behnke

TL;DR

This paper introduces a robust approach for person segmentation and action recognition (walking, waving, sitting) using hemispheric field-of-view LiDAR sensors. By projecting multi-channel LiDAR data into 2D representations and leveraging a modified MaskDINO architecture, the researchers achieve high-precision perception that is both real-time capable and inherently privacy-preserving.

What Problem Does This Paper Address?

For service robots to operate safely in human environments, they must "see" people and understand their activities. While RGB cameras are the standard tool for this, they face two major hurdles: privacy and reliability. Cameras capture facial details that many users find intrusive, and they struggle in low-light or high-contrast conditions.

LiDAR is a promising alternative because it provides precise 3D measurements regardless of lighting and lacks the resolution to "leak" sensitive facial features. However, most LiDAR research focuses on autonomous driving (long-range, narrow vertical FoV). This paper tackles the challenges of hemispheric LiDARs (like the Ouster OSDome), which provide a 180° "bubble" of vision. These sensors have a non-uniform beam distribution—denser near the axis of rotation and sparser at the periphery—which can distort traditional computer vision models.

Methods

The authors propose a pipeline that treats LiDAR data as a multi-channel image rather than a raw point cloud.

  1. Representation: They project the 3D scan into a 512x64 2D matrix where the axes represent the rotation angle and beam number.
  2. Multi-Channel Input: The model uses four standard LiDAR channels: Range, Signal Intensity, Reflectivity, and Near-Infrared (NIR).
  3. Positional Encoding: To help the model understand the sensor's distorted geometry, they add three additional channels representing the Euclidean XYZ coordinates of each point, resulting in a 7-channel input.
  4. Architecture: They adapted MaskDINO, a state-of-the-art transformer-based framework for joint detection and segmentation, using a SwinL (Swin Transformer Large) backbone.
  5. Action Recognition: The model was fine-tuned to not only segment the person but also classify their state into one of three categories: walking, waving, or sitting.

Datasets

The researchers introduced a newly recorded and annotated dataset because existing LiDAR benchmarks (like SemanticKITTI) focus on outdoor automotive scenes.

  • Hardware: Data was collected using an Ouster OSDome-64 mounted on a TIAGo++ omnidirectional mobile robot.
  • Content: 442 scans featuring individuals performing specific actions in indoor environments.
  • Availability: The authors have made the dataset, trained models, and code publicly available to the research community.

Key Insights and Ideas

  • Privacy by Design: The authors argue that by using sensors that don't measure high-resolution imagery in the first place, it becomes impossible to leak private information through model weights or training data.
  • The Power of Positional Encoding: Adding XYZ coordinates as input channels was crucial. It allowed the 2D CNN-based backbone to "understand" the 3D spatial relationships that are warped during the 2D projection of a hemispheric sensor.
  • Channel Contribution: An ablation study revealed that Reflectivity and Range are the most critical channels for person detection. Interestingly, NIR and Signal intensity provided only minor contributions, and in some cases, removing them actually improved performance, suggesting that simpler input representations might be more efficient.
  • Cross-Sensor Generalization: Although trained on a 64-beam sensor, the model performed well on a 128-beam sensor (OSDome-128) without retraining, proving the approach is robust to changes in sensor resolution.

Key Results

  • Segmentation Accuracy: The model achieved an F1-score of 0.97 for person segmentation when the backbone was allowed to train (unfrozen).
  • Action Classification: The system performed exceptionally well across all three actions, with F1-scores consistently above 0.90.
  • Real-Time Performance: On a mobile-grade GPU (Nvidia RTX A4500), the system achieved an inference rate of 11 Hz. This is fast enough for real-time human-robot interaction and tracking.

Limitations and Future Work

The primary limitation is the dataset size; 442 scans is relatively small for deep learning, though the results suggest it was sufficient for these specific tasks. The authors also noted that their "one-layer convolutional encoder" might not be the most efficient way to merge the 7 input channels.

Future work will likely involve exploring more sophisticated encoders to better handle multi-channel data and expanding the library of recognized actions to include more complex human behaviors.

Summary updated

Abstract

Robots need to perceive persons in their surroundings for safety and to interact with them. In this paper, we present a person segmentation and action classification approach that operates on 3D scans of hemisphere field of view LiDAR sensors. We recorded a data set with an Ouster OSDome-64 sensor consisting of scenes where persons perform three different actions and annotated it. We propose a method based on a MaskDINO model to detect and segment persons and to recognize their actions from combined spherical projected multi-channel representations of the LiDAR data with an additional positional encoding. Our approach demonstrates good performance for the person segmentation task and further performs well for the estimation of the person action states walking, waving, and sitting. An ablation study provides insights about the individual channel contributions for the person segmentation task. The trained models, code and dataset are made publicly available.
BibTeX
@inproceedings{seliunina2025person,
  author = {Svetlana Seliunina and Artem Otelepko and Raphael Memmesheimer and Sven Behnke},
  title = {{Person Segmentation and Action Classification for Multi-Channel Hemisphere Field of View LiDAR Sensors}},
  booktitle = {SII},
  year = {2025},
  doi = {10.1109/sii59315.2025.10871131},
}
Figure