Towards Auto-Annotation from Annotation Guidelines: A Benchmark through 3D LiDAR Detection

Yechi Ma, Wei Hua, Shu KongarXiv 2025 · DOI · arXiv

Introduction: The Challenge of High-Octane Data

Annotated data is the lifeblood of machine learning, yet high-quality 3D LiDAR labels are notoriously expensive to produce. Currently, human experts follow meticulous annotation guidelines—textual and visual documents defining object classes—to label datasets. The authors identify a significant gap: while these guidelines train humans, they are rarely used to train machines. Instead, the industry relies on "brute force" supervised learning requiring thousands of manual examples.

The paper proposes Auto-Annotation from Annotation Guidelines, a task where a model "reads" the manual and labels data itself. This is particularly challenging in 3D LiDAR detection due to spatial complexity and the lack of specialized LiDAR foundation models.

The AnnoGuide Benchmark

To address this, the authors created AnnoGuide, a benchmark repurposed from the nuScenes dataset. Unlike standard datasets, AnnoGuide provides only expert guidelines: textual descriptions and 4–8 "iconic" images per class for 18 object categories.

Annotation guidelines for a bicycle, including textual rules and reference images used to instruct human labelers.

No 3D ground-truth cuboids are provided for training, forcing models to rely on zero-shot or few-shot reasoning derived from the guidelines. This setup mimics the real-world scenario where a new sensor or class is introduced, and only a manual exists for guidance.

Methodology: Lifting 2D Intelligence to 3D

The authors' approach uses a pipeline to "lift" 2D foundation model intelligence into 3D space:

  1. Guideline Interpretation: GPT-4o parses textual guidelines to generate optimized descriptive prompts, improving the recall of 2D detectors by using more specific terminology (e.g., "law enforcement officer" instead of just "police").
  2. 2D Detection & Finetuning: GroundingDINO performs open-vocabulary 2D detection, further refined via Multi-Modal Few-Shot Finetuning (MMFF) using the few visual examples provided in the guidelines.
  3. Lifting to 3D: 2D detections are projected into 3D "frustums" within the LiDAR point cloud.
  4. Multi-Hypothesis Testing (MHT): The model generates multiple candidate 3D cuboids, selecting the best fit based on LiDAR point coverage and 2D projection consistency.
  5. Refinement: The Segment Anything Model (SAM) removes background noise, and "future" LiDAR sweeps are aggregated to densify point clouds for more accurate box fitting.

Key Results and Insights

The AnnoGuide pipeline achieves a nearly 2x improvement in 3D detection mean Average Precision (mAP), rising from a baseline of 12.1 to 21.9.

3D annotations generated by the AnnoGuide pipeline, visualized in both RGB and LiDAR views.

Key takeaways from the study include:

  • The LiDAR FM Gap: The heavy reliance on 2D models underscores the urgent need for native LiDAR foundation models to handle spatial data directly.
  • Prompt Engineering: LLM-refined prompts significantly boost 2D detection precision before the 3D lifting process begins.
  • Offline Advantages: Utilizing "future" LiDAR frames allows the model to "see around" objects, improving box accuracy significantly.
  • Sensor Sensitivity: Models trained on one LiDAR sensor often fail on others due to beam pattern differences, making flexible auto-annotation essential for hardware-agnostic scaling.

Conclusion

While AnnoGuide significantly narrows the gap, its best result (21.9 mAP) remains well below fully supervised models (~43.6 mAP). This disparity confirms that auto-annotation from guidelines is a formidable challenge. Future work should focus on instance-level adaptation, contextual reasoning (e.g., recognizing that strollers are typically near adults), and the development of true LiDAR foundation models.

Summary updated

Abstract

Data annotation is crucial for developing machine learning solutions. The current paradigm is to hire ordinary human annotators to annotate data instructed by expert-crafted guidelines. As this paradigm is laborious, tedious, and costly, we are motivated to explore auto-annotation with expert-crafted guidelines. To this end, we first develop a supporting benchmark AutoExpert by repurposing the established nuScenes dataset, which has been widely used in autonomous driving research and provides authentic expert-crafted guidelines. The guidelines define 18 object classes using both nuanced language descriptions and a few visual examples, and require annotating objects in LiDAR data with 3D cuboids. Notably, the guidelines do not provide LiDAR visuals to demonstrate how to annotate. Therefore, AutoExpert requires methods to learn on few-shot labeled images and texts to perform 3D detection in LiDAR data. Clearly, the challenges of AutoExpert lie in the data-modality and annotation-task discrepancies. Meanwhile, publicly-available foundation models (FMs) serve as promising tools to tackle these challenges. Hence, we address AutoExpert by leveraging appropriate FMs within a conceptually simple pipeline, which (1) utilizes FMs for 2D object detection and segmentation in RGB images, (2) lifts 2D detections into 3D using known sensor poses, and (3) generates 3D cuboids for the 2D detections. In this pipeline, we progressively refine key components and eventually boost 3D detection mAP to 25.4, significantly higher than 12.1 achieved by prior arts.
BibTeX
@misc{ma2025towards,
  author = {Yechi Ma and Wei Hua and Shu Kong},
  title = {{Towards Auto-Annotation from Annotation Guidelines: A Benchmark through 3D LiDAR Detection}},
  year = {2025},
  eprint = {2506.02914},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.02914},
}
Figure