Multi-Task Learning with Adapters for Plausibility Prediction: Bridging the Gap or Falling into the Trenches?

Annerose Eichel, Sabine Schulte im WaldeNAACL 2024

TL;DR

This paper investigates whether training a model on multiple related NLP tasks can improve its ability to predict semantic plausibility, a crucial aspect of language understanding. They found that while some knowledge transfer can help, it doesn't substantially improve performance and can even hurt it if the wrong tasks are chosen, especially if the data is imbalanced.

What Problem Does This Paper Address?

Natural language processing (NLP) models are getting better at understanding language, but a key challenge remains: discerning whether a statement or event is plausible or not. This is more than just grammatical correctness; it's about common sense and world knowledge. While existing methods use transformer models, world knowledge features, or lexical abstractions, this paper explores a novel approach: can knowledge gained from training on diverse NLP tasks (like parsing or sentiment analysis) help a model become better at predicting plausibility, especially in scenarios with limited data? They aim to understand which types of knowledge are beneficial and which might be detrimental.

Methods

The researchers employ a multi-task learning framework using "adapters." Adapters are small, task-specific modules that are added to a pre-trained transformer model (like RoBERTa). Instead of fine-tuning the entire large model for each task, only these smaller adapters are trained, making the process more efficient.

They trained 53 adapters across 17 different NLP tasks, ranging from syntactic tasks (like parsing) to lexical semantics (like abstractness prediction) and sentence-level semantics (like question answering). They then used an "adapter-fusion" technique to combine knowledge from these source task adapters with a target adapter trained for plausibility prediction. This allows them to explore how knowledge transfer works in different settings, including varying amounts of training data and class balance.

Datasets

The study utilized four English datasets for plausibility prediction:

  • PEP-3K: A dataset of subject-verb-object events with plausibility judgments.
  • 20Q: A dataset originally for question-style games, adapted for binary plausibility classification.
  • ADEPT: A dataset of English sentence pairs that differ by an adjective, designed to assess changes in plausibility.
  • ELLIE: A dataset focused on elliptical constructions, used here to evaluate generalization to complex linguistic structures.

Key Insights and Ideas

  • Knowledge Transfer is Tricky: Simply adding knowledge from other tasks doesn't automatically make a plausibility model better. The impact depends heavily on the specific source task, the target dataset, and the training data setup.
  • "Wrong" Tasks Hurt: Injecting knowledge from tasks that are not closely related to plausibility can actually degrade performance.
  • Class Imbalance is a Major Hurdle: Plausibility datasets often have a significant imbalance between plausible and implausible examples. This imbalance hinders effective knowledge transfer.
  • Balancing Data Helps, But Not Dramatically: Balancing the training data (ensuring a more even distribution of plausible and implausible examples) can improve performance, but the gains are modest, even if it means using less data overall.
  • Syntactic Tasks are Least Harmful: Among the tested task categories, syntactic tasks showed the least negative impact when their knowledge was transferred to plausibility prediction.

Key Results

The study found that:

  • Single-task adapters can be competitive: In some cases, training a single adapter specifically for plausibility prediction performed as well as or better than multi-task fusion models, especially in low-resource settings.
  • Fusion models didn't offer substantial gains: Across the four datasets and various training configurations, adapter fusion did not lead to significant improvements in plausibility prediction performance compared to the best single-task adapters.
  • Performance varied by dataset and training setting:
    • In low-resource scenarios (PEP-3K, 20Q) and with imbalanced data, knowledge transfer offered minimal benefits.
  • When training on data other than the target dataset's training data, performance often decreased or remained comparable.
  • When using balanced training data, performance was sustained but not substantially improved.
  • Magnitude of change: While some performance changes were statistically significant, the absolute magnitude of these changes was small, typically within +/- 6%.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Context is Key: The current study primarily focuses on sentence-level plausibility. Incorporating more context could potentially resolve ambiguities and improve predictions.
  • Dataset Limitations: The datasets used, while diverse, might not fully capture the nuances of human disagreement on plausibility. Future work could involve collecting more diverse plausibility datasets that account for this disagreement.
  • Model Size: The study used a standard-sized transformer model (RoBERTa-base). Future research could explore whether these findings hold for larger or different model architectures.
  • Graded Plausibility: The study framed plausibility as a binary classification task. Future work could explore graded plausibility, which allows for a more nuanced understanding of how plausible an event is.

Summary updated

BibTeX
@inproceedings{eichel2024multi,
  author = {Annerose Eichel and Sabine Schulte im Walde},
  title = {{Multi-Task Learning with Adapters for Plausibility Prediction: Bridging the Gap or Falling into the Trenches?}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure