3D Annotation-Free Learning by Distilling 2D Open-Vocabulary Segmentation Models for Autonomous Driving

Boyi Sun, Yuhang Liu, Xingxia Wang, Bin Tian, Long Chen, Fei-Yue WangAAAI 2025 · DOI

Abstract

AFOV (Annotation-Free Open-Vocabulary) is a novel framework designed to train 3D point cloud models for autonomous driving without requiring manual labels. By distilling knowledge from advanced 2D open-vocabulary segmentation models and employing a "Tri-Modal" pre-training strategy, AFOV achieves state-of-the-art performance on the nuScenes dataset (47.73% mIoU). The framework addresses common challenges in unsupervised learning, such as label noise and semantic misalignment, through superpoint matching and geometric refinement.

Introduction

In autonomous driving, deep learning models must accurately interpret 3D point clouds from LiDAR sensors. Traditionally, this requires the manual labeling of millions of points—a process that is prohibitively expensive and difficult to scale. While annotation-free learning methods have emerged, they often struggle with noisy labels from 2D models, a lack of semantic understanding in class-agnostic models, and "self-conflict" during training where different parts of the same object are treated as distinct entities.

AFOV bridges this gap by creating a system that learns high-quality 3D representations directly from unannotated data by leveraging the intelligence of 2D image-segmentation models.

Methods

The AFOV framework utilizes a two-stage approach: Tri-Modal Contrastive Pre-training (TMP) and Annotation-Free Training with geometric refinement.

Tri-Modal Contrastive Pre-training (TMP)

Standard pre-training typically aligns images with 3D points. AFOV introduces Text as a third modality, providing a semantic anchor that helps the 3D backbone understand categories (e.g., "pedestrian" vs. "bicycle") more clearly than images alone. To prevent self-conflict, the model employs Superpixel-Superpoint Matching, grouping points into clusters to ensure the model recognizes that different components (like a car door and window) belong to the same semantic object.

The AFOV framework architecture, illustrating the Tri-Modal Pretraining, Annotation-free Training, and Approximate Flat Interaction (AFI) modules.

Annotation-Free Training & Refinement

The model generates pseudo-labels for 3D points using 2D open-vocabulary models such as FC-CLIP or SAN. To mitigate errors in these labels, the authors introduce Approximate Flat Interaction (AFI). This non-parametric module uses 3D geometric priors—specifically the intuition that points on the same flat surface should share a label—to correct "blind spots" and override errors made by the 2D teacher models.

Results and Evaluation

The researchers evaluated AFOV on the nuScenes and SemanticKITTI benchmarks. On the nuScenes annotation-free 3D segmentation task, AFOV reached 47.73% mIoU, outperforming the previous state-of-the-art by 3.13%.

Quantitative results comparing AFOV to existing methods across annotation-free, 1% fine-tuning, and linear probing benchmarks.

Key performance highlights include:

  • Powerful Pre-training: When fine-tuned with only 1% of labeled data, the model reached 51.75% mIoU on nuScenes and 48.14% on SemanticKITTI.
  • Linear Probing: In tests where the 3D backbone was frozen and only a simple classifier was trained, AFOV reached 56.35% mIoU, demonstrating that the backbone learned high-quality features without any human supervision.
  • Efficiency: By pre-generating features rather than running heavy image backbones during 3D training, the actual training phase is highly efficient, requiring only about one hour on a modern GPU.

Limitations and Future Work

Despite setting new records for annotation-free methods, a significant performance gap remains compared to fully supervised models (which achieve ~74.66% mIoU). Additionally, the model's accuracy is currently capped by the performance of its 2D "teacher" models. Future research will focus on closing this supervision gap and expanding the AFOV logic to other critical autonomous driving tasks, such as 3D object detection and trajectory tracking.

Summary updated

BibTeX
@inproceedings{sun2025d,
  author = {Boyi Sun and Yuhang Liu and Xingxia Wang and Bin Tian and Long Chen and Fei-Yue Wang},
  title = {{3D Annotation-Free Learning by Distilling 2D Open-Vocabulary Segmentation Models for Autonomous Driving}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i7.32760},
}
Figure