AnchorAL: Computationally Efficient Active Learning for Large and Imbalanced Datasets

Pietro Lesci, Andreas Vlachos

Abstract

Active learning for imbalanced classification tasks is challenging as the minority classes naturally occur rarely. Gathering a large pool of unlabelled data is thus essential to capture minority instances. Standard pool-based active learning is computationally expensive on large pools and often reaches low accuracy by overfitting the initial decision boundary, thus failing to explore the input space and find minority instances. To address these issues we propose AnchorAL. At each iteration, AnchorAL chooses class-specific instances from the labelled set, or anchors, and retrieves the most similar unlabelled instances from the pool. This resulting subpool is then used for active learning. Using a small, fixed-sized subpool AnchorAL allows scaling any active learning strategy to large pools. By dynamically selecting different anchors at each iteration it promotes class balance and prevents overfitting the initial decision boundary, thus promoting the discovery of new clusters of minority instances. Experiments across different classification tasks, active learning strategies, and model architectures AnchorAL is (i) faster, often reducing runtime from hours to minutes, (ii) trains more performant models, (iii) and returns more balanced datasets than competing methods.

Introduction

Active Learning (AL) is designed to build datasets efficiently by selecting the most informative examples for labeling. However, in massive real-world datasets where minority classes are rare, standard AL faces two major hurdles: the Computational Wall and the Imbalance Trap. The computational wall arises when models must run inference on millions of unlabeled points every iteration. The imbalance trap occurs when AL strategies get stuck refining existing decision boundaries, failing to explore the feature space for isolated clusters of rare data.

Researchers from the University of Cambridge have introduced AnchorAL, a method that makes active learning computationally feasible and effective for highly imbalanced datasets by using a dynamic, anchored subpooling technique.

Methods

AnchorAL introduces an "Anchored Pool Filtering" step before the active learning strategy begins. The process involves:

  1. Anchor Selection: At each iteration, a small number of diverse "anchors" (e.g., 10 per class) are selected from the labeled data using $k$-MEANS++.
  2. Similarity Retrieval: Using a pre-trained encoder and a fast vector index (HNSW), the system retrieves the $K$ nearest unlabeled neighbors for each anchor.
  3. Subpooling: These neighbors form a small, fixed-size subpool (typically ~1,000 instances).
  4. Standard AL: A standard strategy (like Entropy or BADGE) is run only on this tiny subpool.
Conceptual comparison of AnchorAL against standard active learning and SEALS. AnchorAL (bottom) uses anchors to explore new regions, whereas standard AL (top-left) overfits boundaries and SEALS (top-right) suffers from subpool bloat.

By dynamically changing anchors, the method forces the model to explore different regions of the input space, effectively bridging the gap between scattered "islands" of minority data.

Key Results and Insights

AnchorAL was evaluated across four text classification tasks (Amazon-Agri, WikiToxic, etc.) with extreme imbalance (~1% minority) using architectures like BERT and GPT-2.

Performance metrics comparing AnchorAL, RandomSubset, and SEALS. (a) AnchorAL maintains a small, constant subpool size. (b, c) It discovers and annotates a higher percentage of minority instances. (d) It achieves the highest AUC F1 for minority classes.
  • Efficiency: AnchorAL reduced selection time from hours to minutes. In WikiToxic tests, it was approximately 23x faster than the SEALS baseline because it maintains a constant subpool size rather than allowing it to grow linearly.
  • Performance: It consistently achieved higher F1-scores for minority classes compared to random subsetting and standard AL.
  • Data Discovery: The method discovers significantly more minority instances early in the process, resulting in a more balanced and valuable labeled dataset.

Conclusion

AnchorAL breaks the efficiency barrier by prioritizing exploration over exploitation. By anchoring the search to diverse labeled points, it avoids path dependence and scales effectively to massive datasets. Future research may focus on its performance in low-resource languages and its robustness against label noise in real-world annotation scenarios.

Summary updated

Abstract

Active learning for imbalanced classification tasks is challenging as the minority classes naturally occur rarely. Gathering a large pool of unlabelled data is thus essential to capture minority instances. Standard pool-based active learning is computationally expensive on large pools and often reaches low accuracy by overfitting the initial decision boundary, thus failing to explore the input space and find minority instances. To address these issues we propose AnchorAL. At each iteration, AnchorAL chooses class-specific instances from the labelled set, or *anchors*, and retrieves the most similar unlabelled instances from the pool. This resulting *subpool* is then used for active learning. Using a small, fixed-sized subpool AnchorAL allows scaling any active learning strategy to large pools. By dynamically selecting different anchors at each iteration it promotes class balance and prevents overfitting the initial decision boundary, thus promoting the discovery of new clusters of minority instances. Experiments across different classification tasks, active learning strategies, and model architectures AnchorAL is *(i)* faster, often reducing runtime from hours to minutes, *(ii)* trains more performant models, *(iii)* and returns more balanced datasets than competing methods.
BibTeX
@inproceedings{lesci2024anchoral,
  author = {Pietro Lesci and Andreas Vlachos},
  title = {{AnchorAL: Computationally Efficient Active Learning for Large and Imbalanced Datasets}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=2UkMN9DlNX},
}
Figure