Active Learning by Learning

Wei-Ning Hsu, Hsuan-Tien LinAAAI 2015 · OpenReview

Pool-based active learning (AL) is a powerful paradigm for reducing labeling costs by querying the most informative data points from an unlabeled pool. However, selecting the "right" querying strategy is notoriously difficult. Heuristics like uncertainty sampling or density-based clustering often fail to generalize across different datasets—a phenomenon known as the "No Free Lunch" theorem in practice. This paper introduces Active Learning by Learning (ALBL), a framework that dynamically selects and blends active learning strategies on the fly by framing the problem as a multi-armed bandit task.

The Challenge of Strategy Selection

In traditional active learning, researchers rely on fixed, human-designed heuristics. For instance, one might always query the instance the model is most uncertain about. However, the optimal strategy can vary significantly between datasets and even change during different stages of the learning process as the labeled pool grows. While previous attempts have been made to adaptively blend strategies, they often rely on biased heuristics like classification entropy maximization to evaluate performance, which may not correlate with actual test accuracy.

ALBL: A Multi-Armed Bandit Approach

ALBL addresses these challenges by treating different active learning algorithms as "experts" in a multi-armed bandit setting. It utilizes a modified version of the EXP4.P algorithm, designed for adversarial environments, to manage these experts.

Two key innovations drive ALBL's success:

  1. Shared Reward Updates: Unlike standard bandit algorithms that only update the chosen expert, ALBL updates the weights of all experts that suggested the queried data point, accelerating the learning process.
  2. Importance-Weighted Accuracy (IW-ACC): To provide a reliable reward signal without a test set, ALBL uses importance weighting. By including a random sampling strategy in the expert pool, the system can calculate a mathematically unbiased estimator of true test accuracy. This IW-ACC signal allows the bandit to make informed decisions based on predicted performance rather than unreliable heuristics.

Performance against Individual Strategies

The authors evaluated ALBL across six standard UCI datasets: liver, sonar, vehicle, breast, diabetes, and heart. The results demonstrate that no single strategy consistently outperforms others across all tasks. For example, the QUIRE strategy is highly effective on the diabetes dataset, while UNCERTAIN sampling excels on the vehicle dataset.

ALBL performance compared to individual active learning strategies across six UCI datasets.

As shown in Figure 1, ALBL consistently tracks or exceeds the performance of the best underlying individual algorithm for every dataset. In extensive testing, ALBL won or tied against individual experts in the vast majority of checkpoints, proving its robustness as a general-purpose active learning framework.

Comparison with State-of-the-Art Blending

ALBL was also compared against existing blending methods, including COMB (a prior bandit-based method using heuristic rewards) and ALBL-Train (which uses biased training accuracy as a reward).

Comparison of ALBL against state-of-the-art blending methods (COMB) and variants using training accuracy.

Figure 2 highlights the advantage of using the unbiased IW-ACC estimator. On datasets like heart, heuristic-based rewards used by COMB can actively mislead the selection process. ALBL’s mathematically grounded reward signal ensures it remains aligned with true test performance, consistently outperforming both fixed-ratio combinations and prior adaptive state-of-the-art methods.

Conclusion

Active Learning by Learning provides a principled solution to the strategy selection problem. By leveraging a multi-armed bandit framework and an unbiased accuracy estimator, ALBL removes the need for manual heuristic selection. It offers a robust, "plug-and-play" approach that adaptively finds the best mix of strategies for any given dataset, ensuring optimal learning efficiency.

Summary updated

BibTeX
@inproceedings{hsu2015active,
  author = {Wei-Ning Hsu and Hsuan-Tien Lin},
  title = {{Active Learning by Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2015},
  url = {https://openreview.net/forum?id=DLiC_V6P1Fg},
}
Figure