To Label or Not to Label: PALM - A Predictive Model for Evaluating Sample Efficiency in Active Learning Models

Julia Machnio, Mads Nielsen, Mostafa Mehdipour GhaziICCV 2025

TL;DR

This paper introduces PALM, a novel mathematical model that predicts the entire active learning (AL) performance curve using just a few data points. By characterizing AL dynamics with interpretable parameters like coverage efficiency and learning scalability, PALM enables more systematic and cost-effective AL strategy selection, especially in resource-constrained settings.

What Problem Does This Paper Address?

Machine learning models often require vast amounts of labeled data, which can be prohibitively expensive and time-consuming to acquire. Active learning (AL) aims to mitigate this by intelligently selecting the most informative samples for labeling. However, traditional AL evaluation methods typically focus only on final accuracy after a fixed number of labels, failing to capture the full picture of how an AL strategy performs over time. This makes it difficult to compare different AL methods, understand their efficiency, and predict their behavior under varying budget constraints. The paper addresses this gap by proposing a predictive model that characterizes the entire AL learning curve, offering a more nuanced and practical evaluation framework.

Methods

The core contribution is the PALM (Performance Analysis of Active Learning Models) framework. PALM models the AL learning curve using a generalized accuracy function with four key interpretable parameters:

  • $A_{max}$: The maximum achievable accuracy.
  • $\delta$ (Coverage Efficiency): How effectively each labeled sample covers the data space. Higher $\delta$ means better coverage per sample.
  • $\alpha$ (Initial Learning Efficiency): A shifting parameter that accounts for the initial learning progress, especially from unlabeled regions. Lower $\alpha$ means faster initial learning.
  • $\beta$ (Scalability of Learning Gains): Controls the rate at which accuracy improves as more samples are labeled. Higher $\beta$ means steeper accuracy gains.

The model is expressed as a normalized accuracy function that can be fitted to observed AL learning curves using a limited number of data points. This allows for the estimation of these parameters and, consequently, the prediction of the full learning trajectory. The framework is designed to work with or without self-supervised learning (SSL) embeddings.

Datasets

The authors evaluated PALM on the following datasets:

  • CIFAR-10 and CIFAR-100: Standard image classification datasets.
  • ImageNet-50, ImageNet-100, and ImageNet-200: Subsets of the ImageNet dataset, used to evaluate PALM on larger-scale, more complex scenarios.

These datasets were used in both fully supervised settings and with various self-supervised learning (SSL) embeddings (SimCLR, BYOL, MoCov2+, MoCov3).

Key Insights and Ideas

  • Beyond Final Accuracy: The paper strongly argues that focusing solely on final accuracy is insufficient for evaluating AL strategies. PALM's parameters reveal crucial differences in how models learn, not just how well they eventually perform.
  • The Power of Embeddings: Pretrained SSL embeddings significantly boost coverage efficiency ($\delta$) and initial learning efficiency ($\alpha$), leading to faster convergence and better performance, especially in low-budget scenarios.
  • Predictive Power from Partial Data: PALM can accurately predict full AL learning curves using surprisingly few data points (e.g., as few as 1,000 samples for CIFAR-10, representing ~2% of the dataset). This is a game-changer for practical AL deployment.
  • Interpretable Parameters for Strategy Selection: The parameters $\delta$, $\alpha$, and $\beta$ provide concrete metrics for comparing AL strategies. For instance, a strategy with high $\delta$ is more sample-efficient, while one with high $\beta$ scales better with more data.
  • Understanding Trade-offs: PALM helps understand trade-offs between early-stage performance (influenced by $\alpha$) and long-term scalability (influenced by $\beta$ and $A_{max}$), guiding the choice of strategy based on available annotation budgets.

Key Results

  • PALM successfully approximated AL learning curves across various AL strategies (Random Sampling, Uncertainty Sampling, Margin Sampling, etc.) and datasets (CIFAR-10/100, ImageNet subsets).
  • The model demonstrated accurate predictions of full learning dynamics using only a small fraction of the total labeled data (e.g., ~2% for CIFAR-10, ~10% for CIFAR-100).
  • Methods using SSL embeddings consistently showed higher $\delta$ values (e.g., 0.535 for Margin(Emb) vs. 0.094 for Margin without embeddings on CIFAR-10), indicating superior sample efficiency.
  • Non-embedding methods often exhibited higher $\alpha$ values, signifying delayed learning progress compared to embedding-based approaches.
  • For ImageNet-200, MoCov3 embeddings consistently achieved the highest $A_{max}$ and $\delta$, highlighting their superior representation quality.

Limitations and Future Work

  • Deviations in Later Rounds: The paper notes minor deviations in PALM fits during the later stages of learning, particularly on CIFAR-100. This is attributed to the exhaustion of informative samples and the model's simplification of an infinite unlabeled pool. The authors emphasize that PALM is primarily designed for early-stage decision-making.
  • Slow-Starting Methods: For methods with very slow initial learning (like BYOL in some ImageNet experiments), PALM might fit the curve as if it has plateaued if the initial budget is too small to capture the acceleration phase. This suggests a need for sufficient initial data to accurately model such dynamics.
  • Future Work: The authors suggest that PALM lays the groundwork for a new standard in AL evaluation, enabling more systematic and scalable AL design. Future work could involve exploring more complex models or further refining the parameter estimation for edge cases.

Summary updated

BibTeX
@inproceedings{machnio2025label,
  author = {Julia Machnio and Mads Nielsen and Mostafa Mehdipour Ghazi},
  title = {{To Label or Not to Label: PALM - A Predictive Model for Evaluating Sample Efficiency in Active Learning Models}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure