MaChAmp at SemEval-2023 tasks 2, 3, 4, 5, 7, 8, 9, 10, 11, and 12: On the Effectiveness of Intermediate Training on an Uncurated Collection of Datasets.

Rob van der GootACL 2023

TL;DR

This paper investigates the effectiveness of intermediate training for language models across a diverse set of SemEval-2023 NLP tasks. The authors found that while intermediate training can improve performance, the gains are highly dependent on the specific task and the choice of source datasets, and that simply scaling up the number of datasets doesn't guarantee better results.

What Problem Does This Paper Address?

The field of Natural Language Processing (NLP) has seen remarkable progress with the advent of large language models (LLMs). A common practice to adapt these pre-trained models to specific downstream tasks is "intermediate training," where the model is first trained on a related dataset (or set of datasets) before being fine-tuned on the target task. While this approach has shown promise, a key challenge lies in selecting the "right" intermediate tasks and datasets. Previous work often focused on carefully curated datasets or specific types of models (e.g., generative LLMs). This paper aims to explore the effectiveness of intermediate training on a more arbitrary and diverse collection of NLP tasks, specifically all text-based tasks from SemEval-2023, to understand how task selection, dataset diversity, and training strategies impact performance.

Methods

The authors employed the MaChAmp toolkit, using the bert-base-multilingual-cased model as a starting point. They explored several experimental setups:

  • Single-task baselines: Models trained directly on each individual task.
  • Multi-task training: A single model trained on all SemEval-2023 tasks simultaneously.
  • Intermediate training (retrain-all): A multi-task model is trained on all tasks, and then this model is further trained on each target task individually.
  • Intermediate training (retrain-single): A model is trained on a single "best" source task and then fine-tuned on the target task.

They also investigated the impact of:

  • Dataset smoothing: A technique to handle data sparsity.
  • Heterogeneous batching: Combining data from different tasks/datasets within a single batch.

Datasets

The study utilized all text-based tasks from SemEval-2023, encompassing tasks 2 through 12. These tasks cover a wide range of NLP challenges, including:

  • Named Entity Recognition (NER)
  • News Classification and Framing
  • Human Value Classification
  • Clickbait Spoiling
  • Legal Text Understanding
  • Clinical Natural Language Inference
  • Causal Medical Claim Identification
  • Tweet Intimacy Analysis
  • Online Sexism Detection
  • Learning with Disagreements
  • Sentiment Analysis for Low-resource African Languages

The datasets varied significantly in size, language, and task type.

Key Insights and Ideas

  • Task and dataset selection is crucial: Intermediate training doesn't universally improve performance. The effectiveness is highly dependent on the specific target task and the choice of source dataset. Finding the "right" transfer task is non-trivial.
  • "More is not always better": Training on a large, uncurated collection of tasks (multi-task learning) did not consistently outperform finding a single, well-suited source task for intermediate training. Simply scaling up the number of datasets did not lead to robust gains.
  • Heterogeneous batching and smoothing had limited impact: Techniques like dataset smoothing and heterogeneous batching, which have shown promise in other contexts, did not yield consistent performance improvements in this study's setup.
  • Dataset size matters: Correlation studies revealed that dataset size (both source and target) was a significant factor influencing performance gains, with larger datasets sometimes hindering transfer.

Key Results

  • Intermediate training led to performance improvements for most tasks, but the gains were not always substantial.
  • For some tasks, single-task baselines (models trained directly on the target task without intermediate training) performed best.
  • The study found that identifying the best single source task for intermediate training was often more effective than training on a broad mix of tasks.
  • Specific tasks, like task 6-3, 10, 11-1, and 11-2, showed particular benefit from intermediate training.

Limitations and Future Work

  • Uncurated dataset: The use of an arbitrary collection of SemEval tasks, while allowing for a more challenging evaluation, means that the "best" transfer tasks might not have been discoverable within this specific set.
  • Limited tuning: The authors performed minimal task-specific tuning, used no ensembling, and did not incorporate domain-specific embeddings or external data, which could have further boosted performance.
  • Computational cost: The expense of training models, especially for multi-task setups, limited the extent of experimentation.
  • Future work: The authors suggest further investigation into understanding the properties of tasks and datasets that predict successful transfer, and exploring more advanced methods for dataset selection and curriculum learning. They also note that scaling up models might require different strategies than those explored here.

Summary updated

BibTeX
@inproceedings{goot2023machamp,
  author = {Rob van der Goot},
  title = {{MaChAmp at SemEval-2023 tasks 2, 3, 4, 5, 7, 8, 9, 10, 11, and 12: On the Effectiveness of Intermediate Training on an Uncurated Collection of Datasets.}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure