Introduction
Moving Object Segmentation (MOS) is a critical task for autonomous systems, enabling robots to distinguish between dynamic entities like pedestrians or vehicles and static environmental features such as buildings and trees. Accurate MOS is essential for safe navigation, dynamic obstacle avoidance, and high-fidelity mapping. Despite its importance, current MOS research is predominantly limited by datasets captured using homogeneous, mechanically spinning LiDAR sensors (e.g., Velodyne).
As the industry transitions toward heterogeneous sensor suites—incorporating solid-state LiDARs with irregular scanning patterns and varying fields of view—existing models often fail to generalize. To address this, we introduce HeLiMOS, the first dataset providing point-wise MOS labels for heterogeneous LiDAR sensors.
The HeLiMOS Dataset
HeLiMOS fills a significant gap in the perception landscape. While previous datasets like SemanticKITTI or nuScenes have advanced 3D perception, they often lack the combination of multiple heterogeneous sensors and granular, point-wise MOS labels.
The dataset comprises 12,188 labeled point clouds from four distinct sensors:
- Spinning LiDARs: Velodyne VLP-16 (16 channels) and Ouster OS2-128 (128 channels).
- Solid-State LiDARs: Livox Avia (non-repeating flower pattern) and Aeva Aeries II (FMCW-based).
HeLiMOS features a higher "dynamic point ratio" (roughly 2-3%) compared to SemanticKITTI (0.4%), presenting a more challenging and realistic benchmark for urban navigation algorithms.
Automatic Labeling Framework
Labeling 3D point clouds manually is labor-intensive due to data sparsity. To overcome this, we developed a "merging-and-splitting" framework to generate high-quality annotations with minimal human intervention:
- Topology-Based Trajectory Clustering: Grouping robot paths and using submap-to-submap ICP ensures precise scan alignment, mitigating pose drift during loop closures.
- Instance-Aware Initial Annotation: Utilizing ERASOR2, we treat objects as whole instances rather than independent points, leading to cleaner initial labels and avoiding partial deletions.
- Multi-Object Tracking-Based Filtering: Tracking interpolates object positions during brief occlusions and uses bounding box augmentation to correct misclassified static points.
- Human-in-the-loop Refinement: A final manual verification pass ensures "gold standard" quality for the dataset.
Key Insights and Results
Our experiments reveal a significant "sensor gap." Models trained exclusively on spinning LiDAR data perform poorly on solid-state sensors due to the vastly different "traces" left by irregular scanning patterns. For instance, the state-of-the-art 4DMOS model saw its Intersection-over-Union (IoU) drop to just 4.69 when tested on the Velodyne VLP-16 after training on SemanticKITTI.
However, training on the diverse HeLiMOS dataset significantly improves cross-sensor performance. The MapMOS model achieved a mean IoU of 74.72 when trained on the full heterogeneous suite. Furthermore, our automatic labeling framework achieved an IoU of 61.2, nearly tripling the performance of previous tools like Auto-MOS (21.8 IoU).
Conclusion and Future Work
HeLiMOS provides a foundational resource for developing robust, sensor-agnostic MOS algorithms. While sparse sensors like the VLP-16 remain challenging due to pose estimation difficulties, this dataset enables the community to tackle domain generalization across diverse hardware configurations. Future work will focus on developing algorithms that can handle any scanning pattern without extensive retraining.
Summary updated