Introduction
Deep learning for 3D perception, particularly in LiDAR-based autonomous systems, is currently constrained by a significant data bottleneck. While raw point cloud data is relatively easy to collect, manual annotation is prohibitively slow and expensive. This has led to a research landscape heavily skewed towards "urban automotive" environments where large, labeled datasets like SemanticKITTI already exist, leaving other domains—such as agricultural robotics or aerial surveillance—under-explored.
LeAP (Label Any Pointcloud) addresses this challenge by leveraging the zero-shot capabilities of 2D Vision Foundation Models (VFMs), such as Segment Anything (SAM) and Grounding DINO, to automatically generate high-quality 3D semantic labels. By bridging the gap between 2D image segmentation and 3D geometric consistency, LeAP enables the training of 3D models for novel domains without requiring a single manual label.
Methodology
The LeAP framework operates through a three-stage pipeline designed to transform noisy 2D predictions into stable 3D labels.
1. 2D Pseudo-label Generation
The process begins with unlabeled images and a set of class prompts. Grounding DINO identifies potential object bounding boxes, which are then refined by SAM into precise masks. Rather than assigning hard labels, the system generates "soft" labels—probability distributions for each pixel—preserving the uncertainty inherent in the 2D model's predictions.
2. Spatio-Temporal Voxel Fusion
To mitigate the "flickering" and projection errors common in frame-by-frame 2D-to-3D mapping, LeAP projects these probabilities into a 3D voxel grid. A Bayesian update mechanism fuses observations over time, acting as a statistical memory. This approach ensures that transient 2D errors (e.g., a car misidentified as road for a single frame) are smoothed out by consistent historical data, resulting in more stable 3D labels.
3. 3D Consistency Network (3D-CN)
The final stage introduces a "self-correction" loop. A 3D backbone (WaffleIron) is trained on the most confident labels from the voxel grid. By learning the underlying 3D geometry—such as the fact that trees are vertical cylinders—the 3D-CN can correct projection errors that 2D models might miss, such as label "bleeding" through solid objects. These 3D-aware predictions are then fused back into the final voxel grid.
Datasets and Evaluation
LeAP was evaluated on two distinct domains to demonstrate its versatility:
- SemanticKITTI: The standard benchmark for urban automotive LiDAR.
- AgriUAV: A new synthetic dataset introduced in this work, focusing on agricultural scenes (crops, fences, buildings) from a UAV perspective.
Key Results
LeAP demonstrates substantial improvements over standard domain-transfer baselines. On SemanticKITTI, the full LeAP pipeline achieved a 58.1% mIoU, more than doubling the performance of a model pre-trained on a different domain (27.8%).
In the UAV domain, the results were even more striking. Adapting a model to the AgriUAV dataset using LeAP-generated labels resulted in a 34.2 percentage point increase in mIoU compared to an unadapted model. This performance closely approaches that of models trained on human-annotated ground truth, proving that LeAP can effectively "label any pointcloud" in new environments.
Discussion and Limitations
A key insight from this work is that a 3D network trained on noisy labels can actually outperform its own training data by learning geometric priors. Furthermore, the voxel-based approach allows for efficient "retro-active" labeling of points that may have been outside the camera's field of view in previous frames, significantly increasing the density of labeled data.
However, challenges remain. The current voxel grid does not explicitly account for moving objects, which can lead to "ghosting" artifacts. Additionally, the system remains dependent on the quality of the underlying 2D VFMs; consistent misidentifications in 2D can be reinforced by the 3D-CN. Future work will focus on incorporating temporal dynamics to handle moving objects and refining label selection mechanisms.
Summary updated