SemEval 2021 Task 7: HaHackathon, Detecting and Rating Humor and Offense

J. A. Meaney, Steven Wilson, Luis Chiruzzo, Adam Lopez, Walid MagdyACL 2021

TL;DR

This paper introduces HaHackathon, the first shared task to jointly detect humor and offense in text, using a novel dataset of 10,000 tweets and jokes. The results show that while transformer models excel at humor detection, humor controversy remains a significant challenge, highlighting the complex relationship between humor and offense.

What Problem Does This Paper Address?

Humor is a complex and subjective phenomenon that can evoke a wide range of reactions, including offense. While humor detection and offense detection have been studied separately, understanding their interplay is crucial for applications like content moderation and human-computer interaction. This paper addresses the challenge of jointly modeling humor and offense, recognizing that humor can sometimes mask offensive content or be perceived differently across demographic groups. The authors also introduce the concept of "humor controversy," which measures the disagreement among annotators regarding the funniness of a joke, adding another layer of complexity to the problem.

Methods

The HaHackathon task involved three subtasks: binary humor detection, predicting humor and offense ratings (on a 1-5 scale), and a novel humor controversy detection task. Participants were encouraged to use pre-trained language models (PLMs), which proved to be the dominant approach. Many top-performing teams also employed advanced techniques such as task-adaptive training (fine-tuning models on domain-specific data) and adversarial training (making models more robust to small input perturbations). For the humor controversy task, a simple rule-based heuristic was also explored as a baseline.

Datasets

The HaHackathon task utilized a newly created dataset of 10,000 texts, sourced from Twitter (80%) and the Kaggle Short Jokes dataset (20%). This dataset was annotated by 20 individuals from diverse age groups (18-70) for humor intention, general offensiveness, and personal offensiveness. The dataset was split into training, development, and test sets with an 80:10:10 ratio.

Key Insights and Ideas

  • The Intertwined Nature of Humor and Offense: The study found a moderate negative correlation between humor and offense ratings overall, suggesting that humorous texts are often less offensive, and vice-versa. This correlation was stronger for curated joke datasets than for naturally occurring tweets, hinting that the relationship might be more subtle in real-world social media.
  • Humor Controversy is Tricky: Predicting whether a joke will be controversial (i.e., elicit widely varying humor ratings) proved to be a difficult task for most systems. This suggests that annotator disagreement is a complex phenomenon that goes beyond simple humor detection.
  • Transformer Dominance, but Nuances Remain: Pre-trained transformer models, like BERT and RoBERTa, were highly effective for humor and offense detection. However, the analysis of errors revealed that even these powerful models struggled with nuanced forms of humor like irony and sarcasm.
  • Domain Adaptation Matters: Teams that adapted their models to the specific domain of Twitter data often saw performance improvements, underscoring the importance of domain-specific fine-tuning.

Key Results

  • Humor Detection: Top systems achieved high F1 scores (around 0.98) for binary humor detection, indicating strong performance in identifying humorous content.
  • Humor Rating Prediction: The Root Mean Squared Error (RMSE) for humor rating prediction was generally lower for Kaggle texts than for tweets, suggesting that the more structured joke format was easier to rate.
  • Offense Detection: Performance for offense rating prediction was consistently better (lower RMSE) for tweets than for Kaggle texts. This was attributed to the differing distributions of offense ratings, with tweets being more frequently classified as non-offensive.
  • Humor Controversy: This task was the most challenging, with top systems achieving F1 scores around 0.63. Interestingly, a simple rule-based system that labeled texts as controversial if their humor rating exceeded a certain threshold performed competitively, highlighting the difficulty of capturing this nuanced aspect.

Limitations and Future Work

The paper acknowledges that humor controversy is a challenging task, and further research is needed to better understand and model it. The authors also note that while transformer models are powerful, they still struggle with certain types of humor, particularly irony and sarcasm, which can lead to misclassifications. Future work could focus on developing more robust models for these nuanced forms of humor and exploring the relationship between humor and offense in greater detail, especially in diverse cultural contexts. The dataset and annotations are made available for future research.

Summary updated

BibTeX
@inproceedings{meaney2021semeval,
  author = {J. A. Meaney and Steven Wilson and Luis Chiruzzo and Adam Lopez and Walid Magdy},
  title = {{SemEval 2021 Task 7: HaHackathon, Detecting and Rating Humor and Offense}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure