Towards Fine-grained Sampling for Active Learning in Object Detection

Sai Vikas Desai, Vineeth N. BalasubramanianWorkshops @ CVPR 2020 · OpenReview

Abstract

We study the problem of using active learning to reduce annotation effort in training object detectors. Existing efforts in this space ignore the fact that image annotation costs are variable, depending on the number of objects present in a single image. In this regard, we examine a fine-grained sampling based approach for active learning in object detection. Over an unlabeled pool of images, our method aims to selectively pick the most informative subset of bounding boxes (as opposed to full images) to query an annotator. We measure annotation efforts in terms of the number of ground truth bounding boxes obtained. We study the effects of our method on the Feature Pyramid Network and RetinaNet models, and show promising savings in labeling effort to obtain good detection performance.

Introduction

Training state-of-the-art object detectors like Faster R-CNN or RetinaNet typically requires vast amounts of labeled data. However, the process of drawing bounding boxes is significantly more labor-intensive than simple image classification—often by a factor of ten. Traditional Active Learning (AL) strategies for object detection treat images as atomic units: if an image is selected, every object within it must be annotated.

This "all-or-nothing" approach is inefficient for two primary reasons. First, annotation costs are highly variable; an image containing dozens of objects takes much longer to label than a sparse one. Second, many objects in a "rich" image may be redundant or provide little new information to the model. This paper proposes a shift from image-level selection to fine-grained bounding box-level sampling, allowing for more precise control over the annotation budget.

Methodology

The proposed framework moves the unit of selection from the image level to the individual bounding box level. The process follows an iterative loop:

  1. Initial Training: A baseline detector is trained on a small, fully labeled seed set.
  2. Candidate Pool Generation: The baseline model is run on the unlabeled image pool with a low detection threshold to maximize recall, ensuring most objects are captured as candidate boxes.
  3. Fine-grained Sampling: The system selects the $K$ most informative bounding boxes from this pool using metrics such as Mean Classification Uncertainty or Coreset Greedy Selection.
  4. Oracle Annotation: A human annotator reviews the selected boxes, refining them into ground truth labels or marking them as background.
  5. Model Retraining with Partial Labels: To handle images where only some objects are labeled, a dampening parameter ($\lambda_d$) is introduced in the loss function to reduce penalties for potential errors in un-sampled regions.
The fine-grained active learning loop for object detection.

Experimental Results

The researchers evaluated the method on the PASCAL VOC 2012 and MS-COCO datasets using both Feature Pyramid Network (FPN) and RetinaNet architectures.

Efficiency Gains

The fine-grained approach consistently outperformed traditional image-level sampling. On the PASCAL VOC dataset, the proposed method achieved a mean Average Precision (mAP) of 60 using fewer than 6,500 box labels. In contrast, traditional image-based methods required nearly 9,000 labels to reach the same performance level.

Performance comparison on PASCAL VOC 2012 for FPN and RetinaNet.

The 50% Rule

A key finding was that the model could reach approximately 85% of the performance of a fully supervised model (trained on the entire dataset) while using only 50% of the total bounding box labels. This suggests that a significant portion of standard datasets may contain redundant information that does not contribute to model improvement.

Discussion and Conclusion

By focusing on "annotation effort" as the number of bounding boxes rather than images, this work provides a more realistic metric for real-world computer vision projects. Fine-grained sampling allows models to "cherry-pick" difficult or unique objects across a dataset, avoiding the cost of labeling easy or redundant instances.

While promising, the method depends on the initial model's ability to generate a high-recall candidate pool. Future work could involve developing specialized user interfaces for box-level annotation and applying these techniques to high-stakes domains like medical imaging or precision agriculture, where expert time is at a premium.

Summary updated

Abstract

We study the problem of using active learning to reduce annotation effort in training object detectors. Existing efforts in this space ignore the fact that image annotation costs are variable, depending on the number of objects present in a single image. In this regard, we examine a fine-grained sampling based approach for active learning in object detection. Over an unlabeled pool of images, our method aims to selectively pick the most informative subset of bounding boxes (as opposed to full images) to query an annotator. We measure annotation efforts in terms of the number of ground truth bounding boxes obtained. We study the effects of our method on the Feature Pyramid Network and RetinaNet models, and show promising savings in labeling effort to obtain good detection performance.
BibTeX
@inproceedings{desai2020towards,
  author = {Sai Vikas Desai and Vineeth N. Balasubramanian},
  title = {{Towards Fine-grained Sampling for Active Learning in Object Detection}},
  booktitle = {Workshops @ CVPR},
  year = {2020},
  url = {https://openreview.net/forum?id=fjAfofdVRPS},
}
Figure