Amherst685 at SemEval-2021 Task 7: Joint Modeling of Classification and Regression for Humor and Offense

Brian Zylich, Akshay Gugnani, Gabriel Brookman, Nicholas SamorayACL 2021

TL;DR

This paper tackles the challenging task of jointly classifying humor and offense in text, a crucial step for better content moderation and sentiment analysis. The authors found that while intermediate finetuning and backtranslation augmentation didn't improve performance, multitask learning and ensembling different language models significantly boosted results, with their best system ranking 7th in the SemEval-2021 competition.

What Problem Does This Paper Address?

Identifying humor and offense in text is surprisingly difficult. Unlike straightforward sentiment analysis, humor is subjective, culturally influenced, and can even vary within the same person. This makes it a complex problem for NLP systems. The SemEval-2021 Task 7 aimed to address this by not only classifying humor but also predicting its intensity and the offensiveness of the text. This is important for applications like moderating online content, understanding user reviews, and improving conversational AI.

Methods

The authors experimented with several techniques to improve their humor and offense detection models:

  • Transfer Learning: They started with pre-trained language models from the HuggingFace Transformers library, fine-tuning them on the competition dataset.
  • Intermediate Finetuning: They explored fine-tuning models on larger, related datasets (humor and offense detection) before training on the competition data.
  • Backtranslation Augmentation: To increase the size and diversity of the training data, they translated texts to another language and back, hoping to create paraphrases.
  • Multitask Learning: Instead of training separate models for each task (humor classification, humor rating, offense rating), they trained a single model to predict all labels simultaneously. This was achieved by adding multiple prediction heads to the transformer model.
  • Ensembling: They combined the predictions of multiple pre-trained language models (DistilBERT, RoBERTa, BERT, XLNet) to leverage their diverse strengths and mitigate individual weaknesses. They experimented with averaging predictions and selecting the best combination of models.

Datasets

The paper utilized three types of datasets:

  • HaHackathon Competition Dataset: This was the primary dataset for the SemEval-2021 Task 7, containing 8000 texts with labels for humor classification, humor rating, humor controversy, and offense rating.
  • Humor Datasets:
    • 200k Short Texts for Humor Detection: A large dataset of short texts labeled as humorous or not humorous.
  • Reddit Jokes Dataset: A custom-compiled dataset of jokes and other texts scraped from Reddit subreddits like /r/jokes, /r/cleanjokes, /r/reddit.com, and /r/worldnews to provide more diverse negative examples.
  • Offense Datasets:
    • Offensive Language Identification Dataset (OLID): A dataset of 14,200 tweets with binary annotations for offensiveness.
  • Hate Speech and Offensive Language dataset: A dataset of 26,953 tweets with labels indicating how many crowdflower users labeled them as hateful or offensive.

Key Insights and Ideas

  • Intermediate Finetuning and Backtranslation Don't Always Help: Surprisingly, fine-tuning on other humor/offense datasets and using backtranslation augmentation did not improve performance for this specific task. This suggests that the nature of humor, which is highly dependent on specific word choices and cultural context, might not benefit from these standard NLP techniques as much as other tasks.
  • Backtranslation Can Ruin Humor: The paper provides a clear example where backtranslation altered a joke by replacing a key word ("imaginary" with "fantastic"), demonstrating that paraphrasing can easily destroy humor.
  • Multitask Learning is Beneficial: Training a single model to handle multiple related tasks (humor and offense) proved effective, especially for humor rating and controversy prediction.
  • Ensembling is Key: Combining predictions from different language models was a robust strategy, leading to improved overall performance by compensating for individual model limitations.

Key Results

The authors' best performing system achieved a RMSE of 0.5339 on Task 1b (likely referring to a specific subtask of the competition), ranking 7th overall. Table 1 shows that multitask learning combined with ensembling (Multitask +Ens-Best=6LM) yielded the best results across Humor F1/Accuracy, Humor RMSE, Controversy F1/Accuracy, and Offense RMSE.

Limitations and Future Work

The authors acknowledge that their model sometimes struggled with atypical punctuation (like mid-sentence question marks) and that the competition dataset's reliance on tweets for negative examples, compared to their own custom datasets which used news headlines, might have introduced biases. They suggest that the model may have learned to rely on punctuation structure rather than the actual semantic content. Future work could involve addressing these data biases and exploring more sophisticated methods for handling the nuances of humor and offense.

Summary updated

BibTeX
@inproceedings{zylich2021amherst,
  author = {Brian Zylich and Akshay Gugnani and Gabriel Brookman and Nicholas Samoray},
  title = {{Amherst685 at SemEval-2021 Task 7: Joint Modeling of Classification and Regression for Humor and Offense}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure