Abstract
Active Learning (AL) aims to minimize annotation costs by selecting the most informative samples for human labeling. However, traditional uncertainty and diversity-based methods often overlook the local geometry of the data distribution. We present ALLSH (Active Learning guided by Local Sensitivity and Hardness), a strategy that identifies informative data by measuring how sensitive a model's predictions are to small input perturbations. By prioritizing samples where predictions diverge significantly from their augmented counterparts, ALLSH effectively targets regions near decision boundaries. Experimental results across multiple NLP benchmarks demonstrate that ALLSH consistently outperforms standard AL baselines in both in-domain and out-of-domain scenarios, as well as in few-shot prompt selection.
Introduction
In supervised learning, performance is heavily dependent on the volume of high-quality labeled data. Since human annotation is expensive, Active Learning seeks to optimize the selection process. Existing methods typically rely on Uncertainty Sampling (picking low-confidence samples) or Diversity Sampling (picking representative samples). Uncertainty sampling can be prone to selecting outliers or noisy data, while diversity sampling may select redundant, easy-to-learn examples.
ALLSH addresses these gaps by incorporating local sensitivity—the degree to which a prediction changes given a slight input tweak—and learning hardness. By focusing on "sensitive" regions, the model can identify critical examples near the decision boundary that provide the most corrective information.
Methodology: The ALLSH Framework
The ALLSH strategy operates through an iterative loop designed to find the "worst-case" local perturbations for each unlabeled sample.
The Active Learning Loop
- Data Augmentation: For each unlabeled sample $x$, the system generates $K$ augmented copies $x'$ using techniques such as TF-IDF word replacement or backtranslation.
- Measuring Divergence: The original sample and its augmentations are passed through the model. The statistical distance (e.g., KL Divergence) between the original prediction $p_\theta(\cdot|x)$ and augmented predictions $p_\theta(\cdot|x')$ is calculated.
- Worst-Case Augmentation (WCA): ALLSH selects the augmentation that causes the maximum divergence. This "worst-case" scenario identifies the fragility of the model's understanding at that specific point, approximating the local Lipschitz constant of the prediction surface.
- Data Acquisition and Ranking: Samples with the highest worst-case divergence are prioritized for human labeling.
- Curriculum Learning (CL): To further enhance training, the model can be trained on "easier" samples before tackling the "harder" ones identified by the acquisition function.
Experimental Results
The researchers evaluated ALLSH across diverse NLP tasks, including sentiment analysis (SST-2, IMDB), topic classification (AG News), and natural language inference (QNLI, RTE).
Performance Gains
As shown in the results table, ALLSH consistently achieves higher accuracy with smaller annotation budgets. For example, on the SST-2 task with only a 1% annotation budget, ALLSH reached 85.97% accuracy, significantly higher than Random (84.11%) or Entropy (84.53%) sampling. The addition of Curriculum Learning (+ CL) further boosted performance to 86.37%.
Generalization and Prompt Selection
ALLSH demonstrated superior robustness in Out-of-Domain (OD) settings. When transferring a model trained on IMDB to the SST-2 test set, ALLSH maintained a lead of nearly 2% over the best baseline. Furthermore, ALLSH proved highly effective for Prompt Selection in few-shot learning with large language models. On the TREC dataset (8-shot), ALLSH achieved 58.7% accuracy, outperforming the uncertainty-based baseline by 6.5%.
Conclusion and Future Work
ALLSH provides a robust framework for active learning by leveraging local sensitivity as a proxy for decision boundary proximity. While it offers significant accuracy gains and better generalization than traditional methods, it does introduce computational overhead due to the multiple forward passes required for augmentations. Future research may focus on more efficient perturbation methods in discrete text spaces and exploring embedding-based augmentations to further refine the definition of local neighborhoods.
Summary updated