DeepBlueAI at SemEval-2021 Task 7: Detecting and Rating Humor and Offense with Stacking Diverse Language Model-Based Methods

Bingyan Song, Chunguang Pan, Shengguang Wang, Zhipeng LuoACL 2021

TL;DR

This paper presents DeepBlueAI's winning system for SemEval-2021 Task 7, which aims to automatically detect and rate humor and offense in text. They achieved top performance by fine-tuning diverse pre-trained language models (PLMs) like RoBERTa and ALBERT and then stacking their predictions using a novel mechanism, demonstrating the power of model diversity and ensemble methods.

What Problem Does This Paper Address?

Automatically understanding humor and offense in text is a complex AI challenge. It requires not just linguistic understanding but also world knowledge, common sense, and the ability to grasp nuanced relationships between entities. SemEval-2021 Task 7 specifically aimed to tackle this by asking participants to build systems that could:

  1. Detect humor: A binary classification task.
  2. Rate humor: A regression task to quantify how humorous a text is.
  3. Detect controversial humor: A binary classification task for humor that might be considered controversial.
  4. Rate offense: A regression task to quantify how offensive a text is, regardless of whether it's humorous.

This paper focuses on subtask 1a (humor detection) and subtask 2 (offense rating).

Methods

The DeepBlueAI system employs a multi-stage approach centered around pre-trained language models (PLMs):

  • PLM Fine-tuning: They started with two powerful PLMs: ALBERT and RoBERTa. These models were fine-tuned on the task-specific dataset using various hyperparameters and training strategies to create a diverse set of models.
  • Task-Adaptive Pre-training (TAPT): To further adapt the PLMs to the humor and offense domain, they performed an additional pre-training step on the task's dataset itself.
  • Knowledge Distillation (KD): A "teacher-student" approach was used where a larger, well-trained model (teacher) transfers its knowledge to a smaller model (student) by predicting "soft targets" (probabilities) instead of just hard labels. This helps in creating more generalized models.
  • Adversarial Training (AT): This technique enhances model robustness by adding small perturbations to the input embeddings, making the model less sensitive to minor variations and improving its generalization.
  • Pseudo-Labeling (PL): They used their trained models to predict labels for unlabeled data (like the test set) and then incorporated these "pseudo-labels" into the training data to further boost performance.
  • Stacking: The core of their winning strategy is model stacking. They combined the predictions from multiple fine-tuned and augmented PLMs. Instead of a simple average, they used a "validity stacking mechanism" which, in practice, involved averaging predictions from a carefully selected subset of models to achieve the best results. This highlights that not all models contribute equally, and selecting the least correlated ones is key.

Datasets

The primary dataset used is the one provided for SemEval-2021 Task 7. It contains approximately 10,000 samples, split into training, validation, and test sets in an 8:1:1 ratio. The paper notes that the distribution of humor samples is consistent across these sets (around 60% humor). The offense ratings are distributed across a 0-5 scale.

Key Insights and Ideas

  • Diversity is Key: Fine-tuning multiple PLMs with different strategies (TAPT, KD, AT, PL) significantly improves performance compared to using a single model. This diversity helps capture different aspects of humor and offense.
  • Ensembling Works Wonders: Stacking the predictions of these diverse models, even with a simple averaging approach (or selecting the best subset), yields better results than any single model.
  • Adversarial Training Boosts Robustness: Adversarial training consistently improved scores, indicating its effectiveness in making models more robust to variations in text.
  • Not All Models Are Equal: The best ensemble performance wasn't achieved by stacking all models, but by combining a subset. This suggests that selecting models with less correlated predictions is more beneficial.

Key Results

The DeepBlueAI system achieved impressive results:

  • Subtask 1a (Humor Detection): An F1-score of 96.76%, ranking third among all participating teams.
  • Subtask 2 (Offense Rating): An RMSE of 41.2%, ranking first among all participating teams.

Table 2 in the paper details the performance of various combinations of PLMs and training strategies, showing consistent improvements with each added strategy, especially when combining TAPT, PL, and AT.

Limitations and Future Work

  • Subjectivity of Humor/Offense: The paper acknowledges that humor and offense are inherently subjective, influenced by factors like age, gender, and socioeconomic status. While their dataset was balanced, this inherent subjectivity remains a challenge.
  • Ensemble Selection: While they found that combining a subset of models was optimal, the paper doesn't deeply explore how to best select this subset beyond empirical observation.
  • Portability: The authors suggest that a promising future direction would be to test their best-performing system on other humor detection datasets to assess its portability and robustness across different domains.

Summary updated

BibTeX
@inproceedings{song2021deepblueai,
  author = {Bingyan Song and Chunguang Pan and Shengguang Wang and Zhipeng Luo},
  title = {{DeepBlueAI at SemEval-2021 Task 7: Detecting and Rating Humor and Offense with Stacking Diverse Language Model-Based Methods}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure