HeLiMOS: A Dataset for Moving Object Segmentation in 3D Point Clouds From Heterogeneous LiDAR Sensors

Hyungtae Lim, Seoyeon Jang, Benedikt Mersch, Jens Behley, Hyun Myung, Cyrill Stachniss

Introduction

Moving Object Segmentation (MOS) is a critical task for autonomous systems, enabling robots to distinguish between dynamic entities like pedestrians or vehicles and static environmental features such as buildings and trees. Accurate MOS is essential for safe navigation, dynamic obstacle avoidance, and high-fidelity mapping. Despite its importance, current MOS research is predominantly limited by datasets captured using homogeneous, mechanically spinning LiDAR sensors (e.g., Velodyne).

As the industry transitions toward heterogeneous sensor suites—incorporating solid-state LiDARs with irregular scanning patterns and varying fields of view—existing models often fail to generalize. To address this, we introduce HeLiMOS, the first dataset providing point-wise MOS labels for heterogeneous LiDAR sensors.

Visual comparison of point clouds and MOS labels across four heterogeneous LiDAR sensors: Livox Avia, Aeva Aeries II, Velodyne VLP-16, and Ouster OS2-128.

The HeLiMOS Dataset

HeLiMOS fills a significant gap in the perception landscape. While previous datasets like SemanticKITTI or nuScenes have advanced 3D perception, they often lack the combination of multiple heterogeneous sensors and granular, point-wise MOS labels.

Comparison of HeLiMOS with existing odometry and perception datasets. HeLiMOS is unique in providing point-wise MOS labels for heterogeneous, multiple LiDAR setups.

The dataset comprises 12,188 labeled point clouds from four distinct sensors:

  • Spinning LiDARs: Velodyne VLP-16 (16 channels) and Ouster OS2-128 (128 channels).
  • Solid-State LiDARs: Livox Avia (non-repeating flower pattern) and Aeva Aeries II (FMCW-based).

HeLiMOS features a higher "dynamic point ratio" (roughly 2-3%) compared to SemanticKITTI (0.4%), presenting a more challenging and realistic benchmark for urban navigation algorithms.

Automatic Labeling Framework

Labeling 3D point clouds manually is labor-intensive due to data sparsity. To overcome this, we developed a "merging-and-splitting" framework to generate high-quality annotations with minimal human intervention:

  1. Topology-Based Trajectory Clustering: Grouping robot paths and using submap-to-submap ICP ensures precise scan alignment, mitigating pose drift during loop closures.
  2. Instance-Aware Initial Annotation: Utilizing ERASOR2, we treat objects as whole instances rather than independent points, leading to cleaner initial labels and avoiding partial deletions.
  3. Multi-Object Tracking-Based Filtering: Tracking interpolates object positions during brief occlusions and uses bounding box augmentation to correct misclassified static points.
  4. Human-in-the-loop Refinement: A final manual verification pass ensures "gold standard" quality for the dataset.

Key Insights and Results

Our experiments reveal a significant "sensor gap." Models trained exclusively on spinning LiDAR data perform poorly on solid-state sensors due to the vastly different "traces" left by irregular scanning patterns. For instance, the state-of-the-art 4DMOS model saw its Intersection-over-Union (IoU) drop to just 4.69 when tested on the Velodyne VLP-16 after training on SemanticKITTI.

However, training on the diverse HeLiMOS dataset significantly improves cross-sensor performance. The MapMOS model achieved a mean IoU of 74.72 when trained on the full heterogeneous suite. Furthermore, our automatic labeling framework achieved an IoU of 61.2, nearly tripling the performance of previous tools like Auto-MOS (21.8 IoU).

Conclusion and Future Work

HeLiMOS provides a foundational resource for developing robust, sensor-agnostic MOS algorithms. While sparse sensors like the VLP-16 remain challenging due to pose estimation difficulties, this dataset enables the community to tackle domain generalization across diverse hardware configurations. Future work will focus on developing algorithms that can handle any scanning pattern without extensive retraining.

Summary updated

BibTeX
@inproceedings{lim2024helimos,
  author = {Hyungtae Lim and Seoyeon Jang and Benedikt Mersch and Jens Behley and Hyun Myung and Cyrill Stachniss},
  title = {{HeLiMOS: A Dataset for Moving Object Segmentation in 3D Point Clouds From Heterogeneous LiDAR Sensors}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
  doi = {10.1109/iros58592.2024.10801938},
}
Figure