On Dataset Transferability in Active Learning for Transformers

Fran Jelenic, Josip Jukic, Nina Drobac, Jan Snajder

Abstract

Active learning (AL) aims to reduce labeling costs by querying the examples most beneficial for model learning. While the effectiveness of AL for fine-tuning transformer-based pre-trained language models (PLMs) has been demonstrated, it is less clear to what extent the AL gains obtained with one model transfer to others. We consider the problem of transferability of actively acquired datasets in text classification and investigate whether AL gains persist when a dataset built using AL coupled with a specific PLM is used to train a different PLM. We link the AL dataset transferability to the similarity of instances queried by the different PLMs and show that AL methods with similar acquisition sequences produce highly transferable datasets regardless of the models used. Additionally, we show that the similarity of acquisition sequences is influenced more by the choice of the AL method than the choice of the model.

Introduction

Active learning is a "human-in-the-loop" strategy where an acquisition model iteratively selects the most informative unlabeled examples for human labeling. However, this process is inherently biased by the acquisition model's specific inductive bias. This study addresses whether datasets curated via AL remain effective when used to train a different "consumer" model. Given the rapid evolution of NLP models—moving from BERT to RoBERTa to ELECTRA—understanding dataset transferability is crucial for ensuring the long-term value of expensive labeled datasets.

Methodology

The researchers conducted an empirical study involving three popular transformer-based PLMs: BERT, RoBERTa, and ELECTRA. They evaluated three distinct AL strategies:

  • Entropy (ENT): An uncertainty-based method that picks examples the model is most "confused" about.
  • Core-set (CS): A diversity-based method that covers the entire data space.
  • BADGE: A hybrid method combining uncertainty and diversity.

To quantify how differently two models sample data over time, the authors introduced the Acquisition Sequence Mismatch (ASM) metric. This metric uses GloVe embeddings to represent data points and the Hungarian algorithm to find optimal matches between batches, allowing for a precise measurement of sampling similarity across the AL loop.

Evolution of batch and dataset mismatch over active learning steps across different datasets and methods.

Experimental Setup

The study utilized four standard NLP benchmarks for text classification, chosen for their varying complexity and class balance:

  1. Subjectivity (SUBJ): Sentence classification (subjective vs. objective).
  2. CoLA: Linguistic acceptability (grammatical vs. ungrammatical).
  3. AG-News (AGN): Four-class news topic classification.
  4. TREC: Six-class question classification.

Results and Discussion

The most striking finding is that the AL method dictates the sampling sequence much more than the PLM architecture. As shown in the analysis of Acquisition Sequence Mismatch, values are consistently grouped by the AL method across different model pairs (e.g., RoBERTa+BERT, RoBERTa+ELECTRA), regardless of the specific transformer used.

Acquisition Sequence Mismatch (ASM) across different datasets and model pairs, grouped by AL method (ENT, CS, BA).

Key Insights

  • Method Dominance: If the same AL method is used with two different models, they will likely pick very similar data. The choice of strategy is the primary driver of the acquisition sequence.
  • The "Goldilocks" Strategy: BADGE emerged as the most robust method for transfer. By balancing uncertainty and diversity, it creates datasets that are useful for almost any consumer model.
  • Model Flexibility: Transferring data to BERT was generally more successful than to other models, suggesting its pre-training regime might make it more "flexible" to the biases of other models.
  • Transfer Success Rate: Out of 72 transfer configurations, 40 showed positive transfer (outperforming random sampling). A statistically significant negative correlation was found between ASM and transferability ($Δ$AUC), confirming that similar sampling sequences lead to better dataset transfer.

Conclusion

This research demonstrates that datasets acquired via active learning are surprisingly transferable between different transformer-based models, provided the AL strategy is well-chosen. The choice of AL method influences the data selection process far more than the specific architecture of the transformer model itself. Future work will explore these dynamics in larger models and generative tasks.

Summary updated

Abstract

Active learning (AL) aims to reduce labeling costs by querying the examples most beneficial for model learning. While the effectiveness of AL for fine-tuning transformer-based pre-trained language models (PLMs) has been demonstrated, it is less clear to what extent the AL gains obtained with one model transfer to others. We consider the problem of transferability of actively acquired datasets in text classification and investigate whether AL gains persist when a dataset built using AL coupled with a specific PLM is used to train a different PLM. We link the AL dataset transferability to the similarity of instances queried by the different PLMs and show that AL methods with similar acquisition sequences produce highly transferable datasets regardless of the models used. Additionally, we show that the similarity of acquisition sequences is influenced more by the choice of the AL method than the choice of the model.
BibTeX
@inproceedings{jelenic2023dataset,
  author = {Fran Jelenic and Josip Jukic and Nina Drobac and Jan Snajder},
  title = {{On Dataset Transferability in Active Learning for Transformers}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=b5mnKDmsEF},
}
Figure