Instance-wise Supervision-level Optimization in Active Learning

Shinnosuke Matsuo, Riku Togashi, Ryoma Bise, Seiichi Uchida, Masahiro Nomura

TL;DR

This paper introduces Instance-wise Supervision-level Optimization (ISO), a novel active learning framework that dynamically decides not only which data points to label but also how much supervision (full or weak) each point should receive, all within a fixed budget. This adaptive approach significantly boosts annotation efficiency, leading to higher accuracy with lower costs compared to existing methods.

What Problem Does This Paper Address?

Creating large, labeled datasets for machine learning is a major bottleneck, often requiring significant time and expense, especially in specialized domains. Active Learning (AL) aims to mitigate this by intelligently selecting the most informative data points for annotation. However, traditional AL methods typically assume a single level of supervision (e.g., exact labels). This paper tackles the challenge of further reducing annotation costs by incorporating weak supervision – less precise but cheaper labels (like superclasses instead of exact classes) – alongside full supervision. The core problem is how to optimally balance the use of these different supervision levels for each individual data point to maximize learning efficiency under a fixed budget, a problem that existing methods either ignore or address with less granular strategies.

Methods

The proposed framework, Instance-wise Supervision-level Optimization (ISO), introduces a two-pronged approach:

  1. Value-to-Cost Ratio (VCR) Estimation: For each unlabeled instance, ISO calculates a VCR for both full and weak supervision. This VCR quantifies the expected improvement in model accuracy per unit of annotation cost. It's derived from the instance's uncertainty (how much the model is unsure about its prediction) and the expected model improvement, normalized by the respective annotation costs ($C_f$ for full, $C_w$ for weak). Uncertainty is measured using metrics like prediction margin or entropy, and normalized via percentiles to handle outliers. Expected model improvement is estimated by incrementally adding data subsets and observing accuracy changes.
  2. Valuable and Diverse Batch Selection: Instead of simply picking instances with the highest VCR, ISO employs a more sophisticated batch selection strategy. It aims to select a set of instances that are both valuable (high VCR) and diverse. This is achieved by representing each instance as a vector combining its VCR and its feature representation. The selection process then seeks to maximize the "area" enclosed by these vectors, inspired by determinantal point processes (DPPs) and using a k-means++-like seeding algorithm for approximation. This ensures that the selected batch covers a broad range of the data distribution while prioritizing informative points. The algorithm dynamically allocates instances to either the full or weak supervision pool based on the chosen supervision level and its associated cost, updating the budget accordingly.

Datasets

The paper evaluates ISO on two standard image classification datasets:

  • CIFAR100: Contains 100 classes grouped into 20 superclasses. This dataset is used to test the framework's ability to leverage superclass labels as weak supervision.
  • Caltech-UCSD Birds-200-2011 (CUB200): Features 200 bird species, with 70 superclasses derived from class name suffixes. This dataset provides a finer-grained classification task for evaluating ISO.

Key Insights and Ideas

  • Instance-wise Optimization is Crucial: The core insight is that optimizing the supervision level per instance is more effective than global strategies (like fixing a proportion of weak/full supervision for the entire batch). This allows ISO to adapt to the specific characteristics of each data point and the current state of the model.
  • VCR Balances Value and Cost: The VCR metric is a powerful tool for quantifying the cost-effectiveness of different annotation types, enabling informed decisions about where to spend the limited budget.
  • Diversity Prevents Redundancy: Simply picking the "most uncertain" or "highest VCR" points can lead to redundant selections. Incorporating diversity ensures that the selected batch provides broader coverage of the data distribution, leading to more robust learning.
  • Weak Supervision's Impact Scales with Cost: The experiments show that as the cost of weak supervision ($C_w$) decreases relative to full supervision ($C_f$), the benefits of ISO become even more pronounced, highlighting the importance of cost-effective weak labels.

Key Results

  • Superior Accuracy and Cost-Efficiency: ISO consistently outperformed traditional AL methods (Margin, MaxConf, Entropy, Coreset, BADGE) and a state-of-the-art hybrid method (APFWA) that combines full and weak supervision.
  • Significant Budget Savings: On CIFAR100, ISO achieved comparable accuracy to other methods using only three-fifths of the budget, by effectively incorporating weak labels when their cost was half that of full supervision.
  • Adaptability: The approach demonstrated consistent performance improvements across both CIFAR100 and CUB200 datasets, indicating its adaptability to different data characteristics.
  • Ablation Studies Confirm Contributions: Ablation studies showed that both the instance-wise uncertainty component and the diversity-aware batch selection are critical for ISO's superior performance. Removing either component led to a noticeable drop in accuracy.

Limitations and Future Work

  • Classification Focus: The current study is limited to classification tasks. The authors acknowledge that ISO's core concept is applicable to other tasks like object segmentation, where different levels of supervision (e.g., bounding boxes vs. precise masks) can be defined.
  • Two Supervision Levels: The framework currently assumes only two supervision levels (full and weak). Future work could explore extending ISO to handle multiple intermediate supervision levels.
  • Broader Task Exploration: The authors plan to demonstrate the full versatility of ISO by applying it to a wider range of tasks and exploring more complex supervision hierarchies.

Summary updated

Abstract

Active learning (AL) is a label-efficient machine learning paradigm that focuses on selectively annotating high-value instances to maximize learning efficiency. Its effectiveness can be further enhanced by incorporating weak supervision, which uses rough yet cost-effective annotations instead of exact (i.e., full) but expensive annotations. We introduce a novel AL framework, Instance-wise Supervision-Level Optimization (ISO), which not only selects the instances to annotate but also determines their optimal annotation level within a fixed annotation budget. Its optimization criterion leverages the value-to-cost ratio (VCR) of each instance while ensuring diversity among the selected instances. In classification experiments, ISO consistently outperforms traditional AL methods and surpasses a state-of-the-art AL approach that combines full and weak supervision, achieving higher accuracy at a lower overall cost. This code is available at https://github.com/matsuo-shinnosuke/ISOAL.
BibTeX
@inproceedings{matsuo2025instance,
  author = {Shinnosuke Matsuo and Riku Togashi and Ryoma Bise and Seiichi Uchida and Masahiro Nomura},
  title = {{Instance-wise Supervision-level Optimization in Active Learning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  url = {https://openreview.net/forum?id=dM3g2SF4d9},
}
Figure