Few-Shot Point Cloud Region Annotation with Human in the Loop

Siddhant Jain, Sowmya Munukutla, David HeldarXiv 2019 · OpenReview

Abstract

We propose a point cloud annotation framework that employs human-in-loop learning to enable the creation of large point cloud datasets with per-point annotations. Sparse labels from a human annotator are iteratively propagated to generate a full segmentation of the network by fine-tuning a pre-trained model of an allied task via a few-shot learning paradigm. We show that the proposed framework significantly reduces the amount of human interaction needed in annotating point clouds, without sacrificing on the quality of the annotations. Our experiments also suggest the suitability of the framework in annotating large datasets by noting a reduction in human interaction as the number of full annotations completed by the system increases. Finally, we demonstrate the flexibility of the framework to support multiple different annotations of the same point cloud enabling the creation of datasets with different granularities of annotation.

Introduction

As autonomous driving and robotics advance, the demand for high-quality 3D point cloud data grows. However, labeling these datasets is a significant bottleneck: deep learning models require massive amounts of labeled data to learn segmentation, but creating those labels manually is incredibly labor-intensive. Unlike 2D images, 3D point clouds are sparse, massive, and difficult to manipulate via standard 2D interfaces. Existing automatic methods often fail on fine details, while manual methods are too slow to scale. This paper introduces a human-in-the-loop framework that uses few-shot learning and geometric "region growing" to propagate a handful of human clicks into full semantic segmentations, reducing the manual workload by up to 10x.

The Proposed Framework

The framework employs an iterative pipeline that fuses human intuition with algorithmic efficiency to bridge the gap between a single human click and the thousands of points a neural network needs to see to start learning.

The iterative human-in-the-loop annotation pipeline, showing the progression from sparse manual clicks to network prediction and correction.

Methodology

  1. Sparse Manual Annotation: A human annotator selects a few representative points for each semantic part (e.g., the leg of a chair vs. the seat).
  2. Geometric Region Growing: The system uses geometric cues—like surface normals, curvature, and spatial proximity—to automatically expand these few clicks into larger "patches" of labeled points.
  3. Few-Shot Fine-Tuning: The framework takes a pre-trained segmentation model (specifically PointNet pre-trained on ShapeNet) and fine-tunes it using the newly generated sparse patches.
  4. Smoothness Loss: To ensure the model doesn't produce fragmented labels, a custom loss function using Kullback-Leibler (KL) divergence penalizes neighboring points that have different predicted labels, weighted by their physical distance.
  5. Iterative Correction: The model predicts labels for the whole cloud. The human then provides "sparse corrections" on mispredicted areas, and the model fine-tunes again.

Experimental Evaluation

The framework was evaluated using ShapeNet for pre-training and a Large Dataset of Object Scans (Choi et al., 2016) to test performance on reconstructed, noisy, real-world 3D scans.

Key Results

  • Massive Efficiency Gains: In a 3-class chair segmentation task, a naive manual approach required ~117 clicks. The proposed framework achieved the same result with only 12 clicks.
  • Smoothness and Quality: The inclusion of the "smoothness loss" significantly reduced the click count compared to the same model without the loss (e.g., 39 clicks vs. 12 clicks for a 3-class table).
  • Scaling and Flexibility: As more objects from a specific dataset are labeled, the model adapts to that dataset’s specific noise patterns, further reducing the number of human clicks needed. Additionally, the system is flexible regarding granularity; users can choose to segment objects into varying numbers of parts simply by changing the initial clicks.
Example of the annotation process on an airplane model, showing the propagation from sparse points to a full semantic segmentation.

Conclusion and Future Work

The proposed framework demonstrates that human-in-the-loop learning can drastically reduce the effort required for 3D point cloud annotation. While highly effective, the authors acknowledge that manipulating 3D data on a 2D screen remains cognitively taxing. Future work may involve moving the interface into Virtual Reality (VR) to make 3D manipulation more natural and further reduce the time-per-click. Additionally, future studies will look beyond click counts to measure total time and cognitive load as primary metrics for annotation efficiency.

Summary updated

Abstract

We propose a point cloud annotation framework that employs human-in-loop learning to enable the creation of large point cloud datasets with per-point annotations. Sparse labels from a human annotator are iteratively propagated to generate a full segmentation of the network by fine-tuning a pre-trained model of an allied task via a few-shot learning paradigm. We show that the proposed framework significantly reduces the amount of human interaction needed in annotating point clouds, without sacrificing on the quality of the annotations. Our experiments also suggest the suitability of the framework in annotating large datasets by noting a reduction in human interaction as the number of full annotations completed by the system increases. Finally, we demonstrate the flexibility of the framework to support multiple different annotations of the same point cloud enabling the creation of datasets with different granularities of annotation.
BibTeX
@misc{jain2019few,
  author = {Siddhant Jain and Sowmya Munukutla and David Held},
  title = {{Few-Shot Point Cloud Region Annotation with Human in the Loop}},
  year = {2019},
  eprint = {1906.04409},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1906.04409},
}
Figure