Abstract
Recent active learning (AL) approaches in Natural Language Processing (NLP) proposed using off-the-shelf pretrained language models (LMs). In this paper, we argue that these LMs are not adapted effectively to the downstream task during AL and we explore ways to address this issue. We suggest to first adapt the pretrained LM to the target task by continuing training with all the available unlabeled data and then use it for AL. We also propose a simple yet effective fine-tuning method to ensure that the adapted LM is properly trained in both low and high resource scenarios during AL. Our experiments demonstrate that our approach provides substantial data efficiency improvements compared to the standard fine-tuning approach, suggesting that a poor training strategy can be catastrophic for AL.
Introduction
Active Learning (AL) is a powerful strategy for minimizing manual data labeling costs by iteratively selecting the most informative examples for a model to learn from. While most AL research focuses on optimizing acquisition functions—the mathematical criteria for data selection—this paper identifies a critical bottleneck in the training strategy itself.
Modern NLP relies heavily on Pretrained Language Models (PLMs) like BERT. However, standard fine-tuning procedures are typically designed for static, large-scale datasets. In the context of AL, where the labeled dataset starts extremely small and grows incrementally, these standard methods prove unstable. This instability in "few-shot" early iterations can lead to poor performance that even the most sophisticated acquisition functions cannot overcome.
Methods: Making PLMs "AL-Ready"
The authors propose a two-step training suite to stabilize and enhance PLM performance during Active Learning:
- Task-Adaptive Pretraining (TAPT): Before starting the AL process, the model undergoes continued self-supervised pretraining (Masked Language Modeling) using the entire pool of unlabeled task data. This "primes" the model with the specific domain and vocabulary of the target task.
- Robust Fine-Tuning (FT+): The standard fixed-epoch fine-tuning is replaced with a more flexible approach:
- Early Stopping: Models are trained for up to 20 epochs, stopping only when validation loss plateaus.
- Dynamic Warmup: Learning rate warmup is capped at 100 steps or 10% of total steps, preventing counterproductively long warmup periods as the dataset size increases in later AL iterations.
Key Results
The researchers evaluated their approach using BERT-base on five diverse benchmarks: TREC-6, DBPEDIA, IMDB, SST-2, and AGNEWS. The results demonstrate that the choice of training strategy is often more impactful than the choice of acquisition function.
The proposed TAPT-FT+ method achieved full-dataset performance within a 10-15% labeling budget across all datasets. Notably, on the TREC-6 dataset, the method achieved a 45-point accuracy gain over standard fine-tuning after the very first batch of labels (2% of the data). Furthermore, the method improved uncertainty calibration, making uncertainty-based acquisition functions like Entropy significantly more effective.
Analysis of Training Stability
A key finding is the inherent "brittleness" of standard fine-tuning in low-resource regimes. When data is scarce, the typical 3-epoch training rule often fails to reach convergence.
As shown in the analysis of training epochs, increasing the number of optimization steps in the early stages of AL is crucial. The FT+ strategy's use of early stopping naturally adapts to this need, providing more epochs when data is scarce and fewer as it becomes more abundant, thereby ensuring robust performance throughout the AL cycle.
Conclusion and Limitations
This work highlights that a poor training strategy can be catastrophic for Active Learning. By effectively adapting PLMs through TAPT and robust fine-tuning, researchers can achieve massive data efficiency gains.
However, some limitations remain. TAPT can occasionally lead to overfitting on very small or informal datasets. Additionally, while TAPT saves human labeling time, it introduces initial computational overhead. Future work should explore these strategies with larger-scale models like GPT-3 or FLAN to determine if these few-shot characteristics persist at scale.
Summary updated