Combining Active Learning and Task Adaptation with BERT for Cost-Effective Annotation of Social Media Datasets

Jens Lemmens, Walter DaelemansACL 2023

TL;DR

This paper introduces a novel approach that combines active learning (AL) with unsupervised task adaptation using BERT to significantly reduce the cost of annotating social media data for machine learning tasks. By pre-training BERT on unlabeled data before AL, the authors demonstrate that they can achieve performance comparable to full supervised training with a fraction of the annotated data, making advanced NLP models more accessible for resource-constrained projects.

What Problem Does This Paper Address?

Annotating large datasets, especially those from social media, is crucial for training high-performing NLP models like BERT. However, this process is notoriously expensive and time-consuming. While active learning (AL) has been a successful strategy for reducing annotation costs by intelligently selecting data points for labeling, it often struggles to reach the performance of models trained on the entire dataset. This paper addresses the challenge of achieving cost-effective annotation for social media data, aiming to bridge the gap between AL's efficiency and the performance ceiling of full supervised learning.

Methods

The core of this work lies in the synergistic combination of two techniques:

  • Pool-based Active Learning (AL): This is a standard AL strategy where a model, initially trained on a small labeled set, queries the most "uncertain" data points from a large unlabeled pool for annotation. The intuition is that the model learns most from examples it is least confident about.
  • Unsupervised Task Adaptation via Masked Language Modeling (MLM): This technique leverages BERT's pre-training objective. Before the AL process begins, BERT is further trained on the entire unlabeled dataset using MLM. This "adapts" the model to the specific domain and characteristics of the social media data, aiming to improve its understanding and prediction capabilities.

The authors explore two ways of updating the model during the AL process: model re-initialization (training a new model from scratch with each new batch of labeled data) and checkpoint updating (continuing to fine-tune the existing model).

Datasets

The authors evaluated their approach on three datasets:

  • FRENK: A social media dataset containing Facebook comments labeled for hate speech. This dataset presents challenges due to class imbalances and the nuanced nature of hate speech.
  • GoEmotions: A social media dataset derived from Reddit comments, annotated with 28 emotions in a multi-label setting. Its large number of classes and class imbalances make annotation labor-intensive.
  • 20 News Groups: A classic benchmark dataset for text classification, used here to further validate the proposed approach on a non-social media corpus with many fine-grained classes.

Key Insights and Ideas

  • Task Adaptation is a Game-Changer for AL: Pre-training BERT on the target domain data (task adaptation) before starting active learning significantly boosts performance. It seems to make AL's uncertainty sampling more effective.
  • Re-initialization Outperforms Checkpoint Updating: For language models in AL, re-initializing the model with each new batch of data generally leads to more stable and better results compared to incrementally updating the model's checkpoints.
  • AL Can Be Suboptimal Without Adaptation: On certain datasets (like 20 News Groups), standard AL (relying solely on prediction uncertainty) can actually lead to worse performance than random sampling. Task adaptation acts as a crucial "safety net" to counteract this.
  • Focus on Difficult Classes: The most significant improvements from task adaptation and AL are observed in the most challenging or underrepresented classes within the datasets, such as "violent speech" in FRENK.

Key Results

  • Cost-Effective Annotation: The proposed approach achieved performance comparable to the upper-bound baseline (full supervised training) using only 29.7% of the annotated data for FRENK and 54% for 20 News Groups.
  • Significant Performance Gains: On FRENK, the combination of task adaptation and AL with model re-initialization surpassed the upper bound baseline with a fraction of the data, reaching an F1-macro of 59.4% with only 2,500 training instances.
  • GoEmotions Challenges: While the upper bound was not reached on GoEmotions, task adaptation still provided substantial improvements across all AL strategies, with an average F1 improvement of 7.2%.
  • Negative Impact of Standard AL Mitigated: For the 20 News Groups dataset, where standard AL showed a negative effect on performance, task adaptation alone improved results by 2.9%, and when combined with AL, it yielded an average improvement of 2.6% over random sampling.

Limitations and Future Work

  • Model Calibration: The authors note that neural models are often poorly calibrated, meaning their confidence scores (used for uncertainty sampling) might not accurately reflect true prediction uncertainty. Future work could explore alternative metrics for informativeness beyond simple prediction confidence.
  • Limited Scope: The study was conducted on a limited number of tasks and datasets. Further research is needed to assess the approach's generalizability across different domains and tasks.
  • Future Directions: The authors suggest investigating the effect of calibration quality on AL, exploring combinations of AL with other techniques like data augmentation and weak supervision, and applying the approach to a wider range of tasks and datasets. They also propose further research into whether other metrics might be more effective for AL with pre-trained language models.

Summary updated

BibTeX
@inproceedings{lemmens2023combining,
  author = {Jens Lemmens and Walter Daelemans},
  title = {{Combining Active Learning and Task Adaptation with BERT for Cost-Effective Annotation of Social Media Datasets}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure