Abstract
Active learning for imbalanced classification tasks is challenging as the minority classes naturally occur rarely. Gathering a large pool of unlabelled data is thus essential to capture minority instances. Standard pool-based active learning is computationally expensive on large pools and often reaches low accuracy by overfitting the initial decision boundary, thus failing to explore the input space and find minority instances. To address these issues we propose AnchorAL. At each iteration, AnchorAL chooses class-specific instances from the labelled set, or anchors, and retrieves the most similar unlabelled instances from the pool. This resulting subpool is then used for active learning. Using a small, fixed-sized subpool AnchorAL allows scaling any active learning strategy to large pools. By dynamically selecting different anchors at each iteration it promotes class balance and prevents overfitting the initial decision boundary, thus promoting the discovery of new clusters of minority instances. In experiments across different classification tasks, active learning strategies, and model architectures AnchorAL is (i) faster, often reducing runtime from hours to minutes, (ii) trains more performant models, (iii) and returns more balanced datasets than competing methods.
Introduction
Active Learning (AL) is a cornerstone of modern machine learning, designed to minimize annotation costs by intelligently selecting the most informative data points for human labeling. However, as datasets scale to billions of instances and models grow in complexity, traditional AL strategies face significant bottlenecks. In particular, real-world datasets are often highly imbalanced, where the classes of interest are rare and scattered throughout a massive unlabeled pool. This paper introduces AnchorAL, a novel "anchored" pool filtering method that addresses both the computational overhead of large-scale AL and the difficulty of discovering rare minority class instances.
The Challenges: Scalability and Path Dependence
Standard pool-based active learning suffers from two primary limitations when applied to massive, imbalanced datasets:
- Computational Overhead: In every AL iteration, the model must evaluate every single unlabeled instance to determine its informativeness. For web-scale datasets or large language models (LLMs), this evaluation creates a prohibitive "waiting time" for human annotators, stalling the labeling pipeline.
- The "Missed Cluster" Effect: In imbalanced scenarios, minority classes are often isolated in the feature space. Standard AL strategies tend to focus on refining the existing decision boundary—a phenomenon known as path dependence. This narrow focus prevents the model from exploring the wider input space, causing it to miss entire clusters of rare instances that fall outside the initial boundary's immediate vicinity.
Existing attempts to mitigate these issues, such as random subsetting, often fail because they are unlikely to include rare minority instances in the reduced subpool. Other filtering methods, like SEALS, can lead to subpools that grow uncontrollably or become redundant over time.
The AnchorAL Framework
AnchorAL introduces a dynamic filtering step before the active learning selection occurs. Instead of evaluating the entire pool, AnchorAL constructs a small, high-quality subpool using a four-step process:
- Anchor Selection: At the start of each iteration, the system selects a small set of diverse "anchors" from the already labeled data for each class. By using the k-means++ initialization scheme, AnchorAL ensures these anchors are spread across the known feature space rather than clustering in one region.
- Similarity Retrieval: Using a pre-trained, fixed encoder (such as MPNet), the system retrieves the $K$ nearest neighbors for each anchor from the massive unlabeled pool.
- Subpool Formation: These retrieved instances are aggregated into a small, fixed-size subpool (e.g., 1,000 instances). Because the subpool size is constant, the computational cost of the subsequent AL strategy remains predictable and low.
- Active Selection: Any standard AL strategy (e.g., Entropy-based uncertainty or BADGE) is then applied only to this subpool to select the final batch for annotation.
By dynamically rotating anchors each round, the system is forced to "scout" different regions of the data distribution, effectively breaking path dependence and promoting the discovery of new minority clusters.
Experimental Evaluation
The researchers evaluated AnchorAL across four text classification benchmarks modified to simulate extreme imbalance, where the minority class represented only ~1% of the data: Amazon-Agri, Amazon-Multi, WikiToxic, and Agnews-Bus. These tests spanned six model architectures, including BERT, ALBERT, DeBERTa, GPT-2, and T5.
Key Results
- Efficiency Gains: AnchorAL transformed the selection process from a matter of hours to minutes. On the Amazon-Agri task, AnchorAL required only ~4.4 minutes per iteration, compared to ~85.4 minutes for the SEALS baseline.
- Improved Performance: Despite evaluating a fraction of the total pool, AnchorAL consistently achieved higher F1-scores on minority classes. It outperformed random subsetting, which frequently missed rare classes, and SEALS, which struggled with path dependence.
- Data Balance: The method successfully discovered significantly more minority instances than competing strategies, resulting in a more balanced and representative labeled dataset for final model training.
Discussion and Conclusion
The success of AnchorAL lies in its use of semantic similarity as a guide for exploration. By "anchoring" the search to known minority points and ensuring diversity through k-means++, the framework provides a constant-time scaling solution that is model-agnostic. This allows researchers to use expensive, high-performing classifiers without the penalty of re-encoding massive datasets in every iteration.
While the method currently relies on high-quality English embedding models and assumes a perfect labeling oracle, it represents a significant step toward making active learning feasible for the gargantuan, imbalanced datasets common in industrial and web-scale applications. Future work will focus on extending these benefits to multi-lingual contexts and increasing robustness against noisy human labels.
Summary updated