Abstract
Active learning (AL) is a powerful paradigm for reducing labeling costs by identifying the most informative examples for model training. However, the datasets produced by AL are often biased toward the specific model used to select them. This paper investigates the transferability of these datasets across different transformer-based pre-trained language models (PLMs). By evaluating combinations of models like BERT, RoBERTa, and ELECTRA, the study demonstrates that AL gains are generally transferable. The research introduces the Acquisition Sequence Mismatch (ASM) metric to show that the choice of AL strategy has a more significant impact on transferability than the model architecture itself.
Introduction
In the rapidly evolving field of Natural Language Processing (NLP), practitioners often use Active Learning (AL) to minimize the expense of manual data labeling. Instead of labeling a random sample, an "acquisition model" selects instances that maximize its learning efficiency. While effective, this process creates a potential "data bottleneck": the resulting dataset is inherently biased toward the acquisition model's specific weaknesses and inductive biases.
As newer and more efficient Transformer architectures emerge, a critical question arises: is a dataset curated for a model like BERT still optimal for a newer model like ELECTRA? If AL-acquired datasets are strictly model-dependent, their long-term value is diminished. This study explores whether the performance gains of AL persist when the acquisition model and the "consumer model" (the model trained on the final dataset) differ.
Methodology
Experimental Framework
The researchers employed a cross-model evaluation framework involving two distinct roles:
- Acquisition Model: The model used to select informative data points from an unlabeled pool.
- Consumer Model: The model trained on the data selected by the acquisition model to evaluate its performance.
Three prominent Transformer models were utilized: BERT, RoBERTa, and ELECTRA. To ensure a comprehensive analysis, the study tested three diverse AL strategies:
- Entropy Sampling: An uncertainty-based method that selects examples the model is least confident about.
- Core-set: A diversity-based method that aims to select a representative subset of the entire data distribution.
- BADGE (Batch Active learning by Diverse Gradient Embeddings): A hybrid approach that incorporates both model uncertainty and data diversity.
Datasets
The experiments were conducted across four text classification benchmarks:
- Subjectivity (SUBJ): Binary classification of subjective vs. objective text.
- CoLA: The Corpus of Linguistic Acceptability, focusing on grammatical correctness.
- AG-News (AGN): A four-class topic classification dataset.
- TREC: A six-class question-type classification dataset.
Measuring Similarity: Acquisition Sequence Mismatch (ASM)
To quantify how similarly two models perceive data, the authors introduced Acquisition Sequence Mismatch (ASM). This metric uses GloVe embeddings and the Hungarian algorithm to calculate the distance between the sequences of instances selected by different model-method combinations. A lower ASM indicates that two configurations are selecting similar types of information.
Key Findings
1. Transferability is the Norm
The study found that AL gains are remarkably robust. In 40 out of 72 experimental configurations, transferring an AL-acquired dataset resulted in significantly better performance than random sampling (measured by $\Delta$AUC > 0). On simpler tasks like SUBJ, transfer success reached 100%.
2. Strategy Over Architecture
One of the most striking insights is that the AL method influences the resulting dataset more than the model architecture. If two different models (e.g., BERT and RoBERTa) use the same AL strategy (e.g., BADGE), they tend to select highly similar data points. Conversely, using different AL strategies with the same model results in vastly different datasets.
3. The Robustness of Hybrid Methods
BADGE emerged as the most reliable strategy for data transfer. By balancing uncertainty and diversity, BADGE avoids the "greedy" selection of instances that might only be relevant to one specific model's internal state. This makes the resulting datasets more "universally" informative for various consumer models.
4. The Role of Task Complexity
Transferability is sensitive to task difficulty. While simple binary classification tasks showed high transferability, complex tasks like TREC (multi-class and imbalanced) were more prone to "negative transfer," where the AL-selected data occasionally performed worse than random sampling when moved to a different consumer model.
5. ASM and Performance Correlation
The research established a statistically significant negative correlation between ASM and transferability. This confirms that the more similarly two models "see" the data (lower ASM), the more effectively the AL gains will transfer between them.
Conclusion and Future Work
This research provides a positive outlook for NLP practitioners: expensive datasets curated via Active Learning are not necessarily locked to the models that created them. The findings suggest that by prioritizing hybrid AL strategies like BADGE, organizations can build high-quality, transferable datasets that remain valuable even as they upgrade their underlying Transformer architectures.
Future research directions include expanding this analysis to non-English languages and more complex NLP tasks such as Named Entity Recognition (NER) and Question Answering, where the nuances of data selection may differ from standard text classification.
Summary updated