Instance-wise Supervision-level Optimization in Active Learning

Shinnosuke Matsuo, Riku Togashi, Ryoma Bise, Seiichi Uchida, Masahiro Nomura

Abstract

Active learning (AL) is a label-efficient machine learning paradigm that focuses on selectively annotating high-value instances to maximize learning efficiency. Its effectiveness can be further enhanced by incorporating weak supervision, which uses rough yet cost-effective annotations instead of exact (i.e., full) but expensive annotations. We introduce a novel AL framework, Instance-wise Supervision-Level Optimization (ISO), which not only selects the instances to annotate but also determines their optimal annotation level within a fixed annotation budget. Its optimization criterion leverages the value-to-cost ratio (VCR) of each instance while ensuring diversity among the selected instances. In classification experiments, ISO consistently outperforms traditional AL methods and surpasses a state-of-the-art AL approach that combines full and weak supervision, achieving higher accuracy at a lower overall cost.

Introduction

In the era of deep learning, data is abundant, but high-quality labels remain a significant bottleneck due to the high cost of expert annotation. Active Learning (AL) addresses this by identifying the most "informative" data points for labeling. Simultaneously, Weak Supervision (WS) offers a cost-saving alternative by utilizing coarser, less precise labels (e.g., labeling a bird as a "sparrow" rather than a "Song Sparrow").

A critical gap in current research is that AL and WS are often treated as independent choices. Most existing methods either select a batch for full labeling or apply a fixed ratio of weak-to-full labels across the dataset. They fail to consider whether an individual instance's marginal gain justifies its specific annotation cost. This paper introduces Instance-wise Supervision-Level Optimization (ISO), a framework that dynamically determines both which instances to label and how to label them to maximize the utility of a fixed budget.

The ISO Framework

The ISO framework is built upon two core pillars: Value-to-Cost Ratio (VCR) estimation and diverse batch selection.

The ISO framework pipeline: from an unlabeled data pool, it estimates VCRs, selects a diverse batch, and assigns optimal annotation levels.

Value-to-Cost Ratio (VCR) Estimation

For every unlabeled image, ISO calculates the potential "bang for the buck" for both full and weak supervision levels.

  • Value is derived from the model's uncertainty (how confused it is about the image) and the expected model improvement (historical gains from similar labels).
  • Cost is the pre-defined price of the specific label type (e.g., full vs. weak).

By evaluating $v_f(x)$ and $v_w(x)$, the system can identify instances where a cheap weak label provides nearly as much information as an expensive full label, as well as instances where full supervision is essential.

Diverse Batch Selection

Simply selecting instances with the highest VCR can lead to redundancy, such as labeling multiple nearly identical images. ISO incorporates a diversity-aware selection algorithm inspired by k-means++ seeding. It represents each instance in a space that combines its VCR and its feature characteristics, then selects a set that maximizes the "volume" covered in this feature space. This ensures the model learns from a broad and representative sample of the data distribution.

Experimental Results

The authors evaluated ISO on two standard computer vision benchmarks: CIFAR-100 (100 classes, 20 superclasses) and CUB200 (200 bird species, 70 superclasses).

Performance comparison of ISO against conventional AL methods on CIFAR100 and CUB200 datasets.

Key Insights

  • Superior Efficiency: ISO achieved the same accuracy as traditional full-supervision AL methods while using only 60% of the budget.
  • Outperforming State-of-the-Art: ISO consistently outperformed APFWA, the previous state-of-the-art hybrid method. On CIFAR-100, ISO achieved over 10% higher accuracy than standard AL methods at the same budget point.
  • Dynamic Adaptation: Unlike hybrid methods with fixed ratios, ISO's ratio of weak-to-full labels adapts every round based on the model's current state. This "instance-wise" advantage allows the model to use expert labels only where they are truly needed—typically near complex decision boundaries.
  • Robustness: The method remains effective across varying costs for weak labels. As weak labels become cheaper, ISO intelligently shifts more budget toward them to increase the total volume of training data.

Conclusion

ISO demonstrates that treating the level of supervision as a per-instance optimization problem significantly improves the efficiency of active learning. By balancing individual instance value with overall batch diversity, ISO achieves state-of-the-art results while drastically reducing annotation costs. Future work may explore extending this framework to other tasks like object segmentation or multi-level hierarchical labeling.

Summary updated

Abstract

Active learning (AL) is a label-efficient machine learning paradigm that focuses on selectively annotating high-value instances to maximize learning efficiency. Its effectiveness can be further enhanced by incorporating weak supervision, which uses rough yet cost-effective annotations instead of exact (i.e., full) but expensive annotations. We introduce a novel AL framework, Instance-wise Supervision-Level Optimization (ISO), which not only selects the instances to annotate but also determines their optimal annotation level within a fixed annotation budget. Its optimization criterion leverages the value-to-cost ratio (VCR) of each instance while ensuring diversity among the selected instances. In classification experiments, ISO consistently outperforms traditional AL methods and surpasses a state-of-the-art AL approach that combines full and weak supervision, achieving higher accuracy at a lower overall cost.
BibTeX
@inproceedings{matsuo2025instance,
  author = {Shinnosuke Matsuo and Riku Togashi and Ryoma Bise and Seiichi Uchida and Masahiro Nomura},
  title = {{Instance-wise Supervision-level Optimization in Active Learning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  url = {https://openreview.net/forum?id=dM3g2SF4d9},
}
Figure