On the Importance of Effectively Adapting Pretrained Language Models for Active Learning

Katerina Margatina, Loïc Barrault, Nikolaos Aletras

Abstract

Recent active learning (AL) approaches in Natural Language Processing (NLP) proposed using off-the-shelf pretrained language models (LMs). In this paper, we argue that these LMs are not adapted effectively to the downstream task during AL and we explore ways to address this issue. We suggest to first adapt the pretrained LM to the target task by continuing training with all the available unlabeled data and then use it for AL. We also propose a simple yet effective fine-tuning method to ensure that the adapted LM is properly trained in both low and high resource scenarios during AL. Our experiments demonstrate that our approach provides substantial data efficiency improvements compared to the standard fine-tuning approach, suggesting that a poor training strategy can be catastrophic for AL.

Introduction

Active Learning (AL) is a powerful strategy for minimizing manual data labeling costs by iteratively selecting the most informative examples for a model to learn from. While most AL research focuses on optimizing acquisition functions—the mathematical criteria for data selection—this paper identifies a critical bottleneck in the training strategy itself.

Modern NLP relies heavily on Pretrained Language Models (PLMs) like BERT. However, standard fine-tuning procedures are typically designed for static, large-scale datasets. In the context of AL, where the labeled dataset starts extremely small and grows incrementally, these standard methods prove unstable. This instability in "few-shot" early iterations can lead to poor performance that even the most sophisticated acquisition functions cannot overcome.

Methods: Making PLMs "AL-Ready"

The authors propose a two-step training suite to stabilize and enhance PLM performance during Active Learning:

  1. Task-Adaptive Pretraining (TAPT): Before starting the AL process, the model undergoes continued self-supervised pretraining (Masked Language Modeling) using the entire pool of unlabeled task data. This "primes" the model with the specific domain and vocabulary of the target task.
  2. Robust Fine-Tuning (FT+): The standard fixed-epoch fine-tuning is replaced with a more flexible approach:
  3. Early Stopping: Models are trained for up to 20 epochs, stopping only when validation loss plateaus.
  4. Dynamic Warmup: Learning rate warmup is capped at 100 steps or 10% of total steps, preventing counterproductively long warmup periods as the dataset size increases in later AL iterations.

Key Results

The researchers evaluated their approach using BERT-base on five diverse benchmarks: TREC-6, DBPEDIA, IMDB, SST-2, and AGNEWS. The results demonstrate that the choice of training strategy is often more impactful than the choice of acquisition function.

Performance comparison of AL strategies across five benchmarks. The proposed BERT-TAPT-FT+ (solid blue line) consistently outperforms standard fine-tuning baselines (dashed lines), reaching full supervision performance with significantly less data.

The proposed TAPT-FT+ method achieved full-dataset performance within a 10-15% labeling budget across all datasets. Notably, on the TREC-6 dataset, the method achieved a 45-point accuracy gain over standard fine-tuning after the very first batch of labels (2% of the data). Furthermore, the method improved uncertainty calibration, making uncertainty-based acquisition functions like Entropy significantly more effective.

Analysis of Training Stability

A key finding is the inherent "brittleness" of standard fine-tuning in low-resource regimes. When data is scarce, the typical 3-epoch training rule often fails to reach convergence.

Accuracy vs. number of training samples for different epoch counts on AG_NEWS and IMDB. In low-resource settings (e.g., 100 samples), increasing training epochs from 3 to 20 significantly improves accuracy and reduces performance variance.

As shown in the analysis of training epochs, increasing the number of optimization steps in the early stages of AL is crucial. The FT+ strategy's use of early stopping naturally adapts to this need, providing more epochs when data is scarce and fewer as it becomes more abundant, thereby ensuring robust performance throughout the AL cycle.

Conclusion and Limitations

This work highlights that a poor training strategy can be catastrophic for Active Learning. By effectively adapting PLMs through TAPT and robust fine-tuning, researchers can achieve massive data efficiency gains.

However, some limitations remain. TAPT can occasionally lead to overfitting on very small or informal datasets. Additionally, while TAPT saves human labeling time, it introduces initial computational overhead. Future work should explore these strategies with larger-scale models like GPT-3 or FLAN to determine if these few-shot characteristics persist at scale.

Summary updated

Abstract

Recent active learning (AL) approaches in Natural Language Processing (NLP) proposed using off-the-shelf pretrained language models (LMs). In this paper, we argue that these LMs are not adapted effectively to the downstream task during AL and we explore ways to address this issue. We suggest to first adapt the pretrained LM to the target task by continuing training with all the available unlabeled data and then use it for AL. We also propose a simple yet effective fine-tuning method to ensure that the adapted LM is properly trained in both low and high resource scenarios during AL. Our experiments demonstrate that our approach provides substantial data efficiency improvements compared to the standard fine-tuning approach, suggesting that a poor training strategy can be catastrophic for AL.
BibTeX
@inproceedings{margatina2022importance,
  author = {Katerina Margatina and Loïc Barrault and Nikolaos Aletras},
  title = {{On the Importance of Effectively Adapting Pretrained Language Models for Active Learning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=8SrftJdkhA},
}

Loading related content...

Figure