ALISE: Annotation-Free LiDAR Instance Segmentation for Autonomous Driving

Yongxiang Lyu, Guangfeng Jiang, Hongsi Liu, Jun LiuarXiv 2025 · DOI · arXiv

Abstract: The manual annotation of outdoor LiDAR point clouds for instance segmentation is extremely costly and time-consuming. Current methods attempt to reduce this burden but still rely on some form of human labeling. To completely eliminate this dependency, we introduce ALISE, a novel framework that performs LiDAR instance segmentation without any annotations. The central challenge is to generate high-quality pseudo-labels in a fully unsupervised manner. Our approach starts by employing Vision Foundation Models (VFMs), guided by text and images, to produce initial pseudo-labels. We then refine these labels through a dedicated spatio-temporal voting module, which combines 2D and 3D semantics for both offline and online optimization. To achieve superior feature learning, we further introduce two forms of semantic supervision: a set of 2D prior-based losses that inject visual knowledge into the 3D network, and a novel prototype-based contrastive loss that builds a discriminative feature space by exploiting 3D semantic consistency. This comprehensive design results in significant performance gains, establishing a new state-of-the-art for unsupervised 3D instance segmentation. Remarkably, our approach even outperforms MWSIS, a method that operates with supervision from ground-truth (GT) 2D bounding boxes by a margin of 2.53% in mAP (50.95% vs. 48.42%).

Introduction

LiDAR instance segmentation is a cornerstone of perception for autonomous vehicles, enabling the precise identification and localization of individual objects in 3D space. However, the utility of these systems is often bottlenecked by the requirement for massive amounts of manually annotated 3D point cloud data. Labeling a single LiDAR frame can take hours of human effort, making large-scale deployment and adaptation to new environments prohibitively expensive. ALISE (Annotation-Free LiDAR Instance Segmentation) addresses this by introducing a fully unsupervised framework that distills knowledge from 2D Vision Foundation Models (VFMs) into a 3D LiDAR network.

Methodology

The ALISE framework operates in three distinct stages to bridge the gap between 2D image semantics and 3D point cloud geometry.

  1. Unsupervised Pseudo-label Generation (UPG): The process begins by leveraging the zero-shot capabilities of 2D VFMs. GroundingDINO is used for text-prompted object detection, and the Segment Anything Model (SAM) generates high-quality 2D masks. These masks are then projected into 3D space using camera-LiDAR calibration to create initial 3D coarse labels.
The ALISE pipeline for unsupervised pseudo-label generation and offline refinement using VFM priors.
  1. Spatio-Temporal Refinement: To handle the noise inherent in 2D-to-3D projection, ALISE employs a dual refinement strategy. Offline Refinement (OFR) uses a Voxel-Based Semantic Voting (VSV) mechanism that aggregates information across multiple frames and viewpoints, effectively filtering out transient errors. Online Refinement (ONR) utilizes a teacher-student paradigm where an exponential moving average (EMA) of the model provides stable targets during training.
  2. Multi-Faceted Supervision: Rather than relying on "hard" pseudo-labels, ALISE introduces VFM Priors-based Distillation (VPD), which uses soft probability distributions to guide the network, preserving semantic uncertainty. Additionally, a Prototype-based Contrastive Loss (PCL) is implemented to ensure that features belonging to the same semantic class are clustered together in the embedding space, enhancing the model's discriminative power.

Experimental Results

ALISE was evaluated on the Waymo Open Dataset (WOD) and nuScenes. On the Waymo dataset, the unsupervised ALISE model achieved a mean Average Precision (mAP) of 50.95%. This result is particularly significant as it outperforms MWSIS (48.42%), a method that requires ground-truth 2D bounding box supervision.

Performance comparison of ALISE against various supervision methods and the fully supervised baseline.

As shown in the figure above, ALISE demonstrates exceptional data efficiency. When fine-tuned with only 1.2% of labeled data, the model achieves a performance of 59.36% mAP, surpassing the fully supervised baseline (59.26%) trained on 100% of the data. When fully fine-tuned with 100% labels, ALISE reaches 62.30% mAP, a substantial improvement over the standard baseline. This suggests that the unsupervised pre-training provided by ALISE creates a highly robust feature representation that is superior to starting from scratch with full supervision.

Conclusion

ALISE represents a significant step toward truly autonomous perception systems. By eliminating the need for manual 3D annotations and outperforming existing weakly supervised benchmarks, it demonstrates the viability of using 2D foundation models to bootstrap 3D understanding. Future work will focus on reducing the dependency on precise sensor calibration and expanding the framework to handle a wider variety of rare object classes.

Summary updated

Abstract

The manual annotation of outdoor LiDAR point clouds for instance segmentation is extremely costly and time-consuming. Current methods attempt to reduce this burden but still rely on some form of human labeling. To completely eliminate this dependency, we introduce ALISE, a novel framework that performs LiDAR instance segmentation without any annotations. The central challenge is to generate high-quality pseudo-labels in a fully unsupervised manner. Our approach starts by employing Vision Foundation Models (VFMs), guided by text and images, to produce initial pseudo-labels. We then refine these labels through a dedicated spatio-temporal voting module, which combines 2D and 3D semantics for both offline and online optimization. To achieve superior feature learning, we further introduce two forms of semantic supervision: a set of 2D prior-based losses that inject visual knowledge into the 3D network, and a novel prototype-based contrastive loss that builds a discriminative feature space by exploiting 3D semantic consistency. This comprehensive design results in significant performance gains, establishing a new state-of-the-art for unsupervised 3D instance segmentation. Remarkably, our approach even outperforms MWSIS, a method that operates with supervision from ground-truth (GT) 2D bounding boxes by a margin of 2.53% in mAP (50.95% vs. 48.42%).
BibTeX
@misc{lyu2025alise,
  author = {Yongxiang Lyu and Guangfeng Jiang and Hongsi Liu and Jun Liu},
  title = {{ALISE: Annotation-Free LiDAR Instance Segmentation for Autonomous Driving}},
  year = {2025},
  eprint = {2510.05752},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.05752},
  howpublished = {arXiv},
}
Figure