On Dataset Transferability in Active Learning for Transformers

Fran Jelenic, Josip Jukic, Nina Drobac, Jan Snajder

Abstract

Active learning (AL) is a powerful paradigm for reducing labeling costs by identifying the most informative examples for model training. However, the datasets produced by AL are often biased toward the specific model used to select them. This paper investigates the transferability of these datasets across different transformer-based pre-trained language models (PLMs). By evaluating combinations of models like BERT, RoBERTa, and ELECTRA, the study demonstrates that AL gains are generally transferable. The research introduces the Acquisition Sequence Mismatch (ASM) metric to show that the choice of AL strategy has a more significant impact on transferability than the model architecture itself.

Introduction

In the rapidly evolving field of Natural Language Processing (NLP), practitioners often use Active Learning (AL) to minimize the expense of manual data labeling. Instead of labeling a random sample, an "acquisition model" selects instances that maximize its learning efficiency. While effective, this process creates a potential "data bottleneck": the resulting dataset is inherently biased toward the acquisition model's specific weaknesses and inductive biases.

As newer and more efficient Transformer architectures emerge, a critical question arises: is a dataset curated for a model like BERT still optimal for a newer model like ELECTRA? If AL-acquired datasets are strictly model-dependent, their long-term value is diminished. This study explores whether the performance gains of AL persist when the acquisition model and the "consumer model" (the model trained on the final dataset) differ.

Methodology

Experimental Framework

The researchers employed a cross-model evaluation framework involving two distinct roles:

  1. Acquisition Model: The model used to select informative data points from an unlabeled pool.
  2. Consumer Model: The model trained on the data selected by the acquisition model to evaluate its performance.

Three prominent Transformer models were utilized: BERT, RoBERTa, and ELECTRA. To ensure a comprehensive analysis, the study tested three diverse AL strategies:

  • Entropy Sampling: An uncertainty-based method that selects examples the model is least confident about.
  • Core-set: A diversity-based method that aims to select a representative subset of the entire data distribution.
  • BADGE (Batch Active learning by Diverse Gradient Embeddings): A hybrid approach that incorporates both model uncertainty and data diversity.

Datasets

The experiments were conducted across four text classification benchmarks:

  • Subjectivity (SUBJ): Binary classification of subjective vs. objective text.
  • CoLA: The Corpus of Linguistic Acceptability, focusing on grammatical correctness.
  • AG-News (AGN): A four-class topic classification dataset.
  • TREC: A six-class question-type classification dataset.

Measuring Similarity: Acquisition Sequence Mismatch (ASM)

To quantify how similarly two models perceive data, the authors introduced Acquisition Sequence Mismatch (ASM). This metric uses GloVe embeddings and the Hungarian algorithm to calculate the distance between the sequences of instances selected by different model-method combinations. A lower ASM indicates that two configurations are selecting similar types of information.

Key Findings

1. Transferability is the Norm

The study found that AL gains are remarkably robust. In 40 out of 72 experimental configurations, transferring an AL-acquired dataset resulted in significantly better performance than random sampling (measured by $\Delta$AUC > 0). On simpler tasks like SUBJ, transfer success reached 100%.

2. Strategy Over Architecture

One of the most striking insights is that the AL method influences the resulting dataset more than the model architecture. If two different models (e.g., BERT and RoBERTa) use the same AL strategy (e.g., BADGE), they tend to select highly similar data points. Conversely, using different AL strategies with the same model results in vastly different datasets.

3. The Robustness of Hybrid Methods

BADGE emerged as the most reliable strategy for data transfer. By balancing uncertainty and diversity, BADGE avoids the "greedy" selection of instances that might only be relevant to one specific model's internal state. This makes the resulting datasets more "universally" informative for various consumer models.

4. The Role of Task Complexity

Transferability is sensitive to task difficulty. While simple binary classification tasks showed high transferability, complex tasks like TREC (multi-class and imbalanced) were more prone to "negative transfer," where the AL-selected data occasionally performed worse than random sampling when moved to a different consumer model.

5. ASM and Performance Correlation

The research established a statistically significant negative correlation between ASM and transferability. This confirms that the more similarly two models "see" the data (lower ASM), the more effectively the AL gains will transfer between them.

Conclusion and Future Work

This research provides a positive outlook for NLP practitioners: expensive datasets curated via Active Learning are not necessarily locked to the models that created them. The findings suggest that by prioritizing hybrid AL strategies like BADGE, organizations can build high-quality, transferable datasets that remain valuable even as they upgrade their underlying Transformer architectures.

Future research directions include expanding this analysis to non-English languages and more complex NLP tasks such as Named Entity Recognition (NER) and Question Answering, where the nuances of data selection may differ from standard text classification.

Summary updated

Abstract

Active learning (AL) aims to reduce labeling costs by querying the examples most beneficial for model learning. While the effectiveness of AL for fine-tuning transformer-based pre-trained language models (PLMs) has been demonstrated, it is less clear to what extent the AL gains obtained with one model transfer to others. We consider the problem of transferability of actively acquired datasets in text classification and investigate whether AL gains persist when a dataset built using AL coupled with a specific PLM is used to train a different PLM. We link the AL dataset transferability to the similarity of instances queried by the different PLMs and show that AL methods with similar acquisition sequences produce highly transferable datasets regardless of the models used. Additionally, we show that the similarity of acquisition sequences is influenced more by the choice of the AL method than the choice of the model.
BibTeX
@inproceedings{jelenic2023dataset,
  author = {Fran Jelenic and Josip Jukic and Nina Drobac and Jan Snajder},
  title = {{On Dataset Transferability in Active Learning for Transformers}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=b5mnKDmsEF},
}
Figure