Limitations of Active Learning With Deep Transformer Language Models

Mike D'Arcy, Doug DowneyICLR 2022 · OpenReview

TL;DR

Active learning (AL), a technique to reduce labeling costs, struggles with modern large transformer language models, often underperforming random selection. This paper reveals that training instability, not just "unlearnable" outliers, is a key culprit, and suggests mitigating it with multi-model training can improve performance.

What Problem Does This Paper Address?

Training state-of-the-art natural language processing (NLP) models, especially those based on large transformer architectures, requires vast amounts of labeled data. Acquiring these labels is expensive and time-consuming. Active learning (AL) aims to alleviate this by intelligently selecting the most informative unlabeled data points for human annotation. While AL has shown success with smaller NLP models, its effectiveness with the current generation of massive pre-trained transformers (like BERT and RoBERTa) is unclear. Previous studies have yielded conflicting results, with some suggesting AL is beneficial and others finding it ineffective. This paper investigates this discrepancy, specifically focusing on whether AL's limitations persist with larger, more capable transformer models and more challenging NLP tasks.

Methods

The authors explore four active learning strategies:

  • Random: A baseline where data points are selected randomly.
  • Entropy: Selects data points with high uncertainty in the model's output distribution.
  • BALD-MC (Bayesian Active Learning by Disagreement): Uses Monte Carlo dropout to estimate model uncertainty.
  • BatchBALD-MC: A batch-aware version of BALD, aiming to select diverse and informative batches.
  • Coreset: Selects data points that are diverse and representative of the unlabeled data pool.

They fine-tune three pre-trained transformer models: BERT-base, RoBERTa-large, and RoBERTa-base across ten diverse datasets. A key aspect of their experimental setup is evaluating performance using the Area Under the Curve (AUC) of the learning curve, which captures performance across different amounts of labeled data. They also investigate potential causes for AL's failures, including collective outliers and training instability, and propose mitigation strategies like outlier pruning and multi-model training.

Datasets

The paper uses a variety of datasets across two main tasks:

  • Text Classification:
    • AGN (Zhang et al., 2015)
    • DBpedia (Zhang et al., 2015)
    • Movie review sentence polarity (MRP) (Pang & Lee, 2005)
    • Movie review subjectivity (MRS) (Pang & Lee, 2004)
    • Subsampled versions of AGN and DBpedia (AGN-SB-c, DBpedia-SB-c) for binary classification.
  • Multiple-Choice Commonsense Reasoning:
    • aNLI (Bhagavatula et al., 2020)
    • CODAH (Chen et al., 2019)
    • CommonsenseQA (CSQA) (Talmor et al., 2019)
    • HellaSWAG (Zellers et al., 2019)
    • PIQA (Bisk et al., 2020)
    • SWAG (Zellers et al., 2018)

The authors also create text classification versions of the multiple-choice datasets to isolate the effect of question format.

Key Insights and Ideas

  • AL struggles with large transformers: Contrary to some prior work, AL methods often underperform random selection when applied to large pre-trained transformer models, especially as model size increases.
  • Instability is a major culprit: Beyond the known issue of "unlearnable" collective outliers, the paper identifies training instability as a significant factor. AL can select informative examples that, paradoxically, destabilize the training process, leading to lower average performance.
  • Multi-model training as a solution: A simple yet effective strategy to combat instability is training multiple models for each AL selection and choosing the best performer on a validation set. This approach yields performance comparable to outlier pruning but with higher absolute gains.
  • Outlier pruning's mixed success: While outlier pruning (as proposed by Karamcheti et al., 2021) can help, its effectiveness is dataset-dependent, and it can sometimes reduce absolute performance by removing useful examples.
  • Pretraining's complex role: The degree of pretraining has inconsistent effects on AL, with larger models (like RoBERTa) showing more pronounced instability issues compared to smaller ones (like BERT).

Key Results

  • For RoBERTa-base, AL methods often performed worse than random selection, with 10 out of 44 results showing significant degradation.
  • With RoBERTa-large, AL's performance was substantially worse than random across most datasets, with an average difference of -1.7 for BatchBALD vs. Random, compared to +0.4 for RoBERTa-base.
  • The multi-model training approach improved BatchBALD's performance relative to random on datasets where it previously underperformed, and also increased absolute performance.
  • Outlier pruning improved AL's relative performance on some datasets but worsened it on others, and often reduced absolute performance.
  • Ablation studies showed that annotation batch size and adversarial filtering did not significantly explain AL's failures.

Limitations and Future Work

  • Computational Cost: Experiments with larger models and more extensive multi-training were limited by high computational costs, suggesting that further investigation into optimization techniques is needed.
  • Generalizability: While the study covers three models and ten datasets, testing a wider range of models and sizes would further confirm the generality of the findings.
  • Minimum Label Budget: The paper highlights the need to determine the minimum viable label budget for AL to be effective, especially when labels are costly.
  • Instability Mitigation: Exploring advanced techniques to mitigate training instability in large transformers, potentially inspired by recent work on layer normalization placement, is a promising avenue.
  • Developing Robust AL: The ultimate goal is to develop AL methods and optimization techniques that are robust to the challenges posed by large transformer models and complex datasets.

Summary updated

Abstract

Active Learning (AL) has the potential to reduce labeling cost when training natural language processing models, but its effectiveness with the large pretrained transformer language models that power today's NLP is uncertain. We present experiments showing that when applied to modern pretrained models, active learning offers inconsistent and often poor performance. As in prior work, we find that AL sometimes selects harmful "unlearnable" collective outliers, but we discover that some failures have a different explanation: the examples AL selects are informative but also increase training instability, reducing average performance. Our findings suggest that for some datasets this instability can be mitigated by training multiple models and selecting the best on a validation set, which we show impacts relative AL performance comparably to the outlier-pruning technique from prior work while also increasing absolute performance. Our experiments span three pretrained models, ten datasets, and four active learning approaches.
BibTeX
@inproceedings{darcy2022limitations,
  author = {Mike D'Arcy and Doug Downey},
  title = {{Limitations of Active Learning With Deep Transformer Language Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=Q8OjAGkxwP5},
}
Figure