ES-JUST at SemEval-2021 Task 7: Detecting and Rating Humor and Offensive Text Using Deep Learning

Emran Al Bashabsheh, Sanaa Abu AlasalACL 2021

TL;DR

This paper tackles the challenging task of automatically detecting and rating humor and offensive content in text. The authors leverage powerful pre-trained deep learning models, particularly RoBERTa, to achieve strong performance across four distinct sub-tasks, demonstrating the effectiveness of these models for nuanced language understanding.

What Problem Does This Paper Address?

Understanding and generating human language is a cornerstone of Natural Language Processing (NLP). However, language is rich with subtleties, and humor, in particular, presents a significant challenge. Humor relies on wordplay, cultural context, and subjective interpretation, making it difficult for machines to grasp. Furthermore, humor can sometimes border on or directly intersect with offensive content, creating a complex landscape for automated analysis. This paper addresses SemEval-2021 Task 7, which aims to build systems capable of not only identifying whether a text is humorous but also rating its degree of humor, determining if it's controversial, and assessing its offensiveness. This is crucial for applications like content moderation, social media analysis, and building more sophisticated conversational agents.

Methods

The core of this research lies in applying and evaluating pre-trained transformer-based language models. The team experimented with several prominent models:

  • BERT: A foundational model for understanding contextual text.
  • RoBERTa: An optimized version of BERT, trained on a larger dataset with a modified training procedure.
  • XLM-RoBERTa: A multilingual variant of RoBERTa, designed for cross-lingual understanding.

As a baseline, they also employed a BERT embedding layer followed by a Bi-LSTM (Bidirectional Long Short-Term Memory) network. This hybrid approach leverages the contextual understanding of BERT embeddings and the sequential processing capabilities of LSTMs.

The models were fine-tuned for four specific sub-tasks:

  1. Is-Humor: A binary classification task to determine if text is humorous (yes/no).
  2. Humor-Rating: A regression task to predict the degree of humor on a scale of 0-5.
  3. Humor-Controversy: A binary classification task to determine if humorous text is controversial.
  4. Offense-Rating: A regression task to predict the degree of offensiveness on a scale of 0-5.

Datasets

The research utilized a dataset specifically curated for SemEval-2021 Task 7. This dataset consists of texts annotated with four categories: is-humor, humor-rating, humor-controversy, and offense-rating. Each text was labeled by 20 annotators. For the "is-humor" and "humor-controversy" tasks, majority voting was used. For "humor-rating" and "offense-rating," the average of the annotators' ratings was taken. A crucial preprocessing step involved removing "NaN" values, which arose when a text was not classified as humorous, making the humor-related ratings irrelevant. The paper provides a breakdown of the training, development, and testing set sizes for each category.

Key Insights and Ideas

  • RoBERTa Reigns Supreme: The RoBERTa model consistently outperformed other approaches across all four sub-tasks, highlighting its robustness and effectiveness in capturing the nuances of humor and offense.
  • Pre-trained Models are Powerful: The success of RoBERTa, BERT, and XLM-RoBERTa underscores the significant advantage of leveraging large-scale pre-trained models for complex NLP tasks, even those involving subjective human judgments like humor.
  • Subjectivity of Humor: The paper implicitly acknowledges the inherent subjectivity of humor, as evidenced by the need for multiple annotators and the presence of "NaN" values when humor isn't detected. This makes achieving perfect scores challenging.
  • Interconnectedness of Tasks: The sub-tasks are designed to be related, with humor-rating and controversy detection building upon the initial identification of humor. This hierarchical structure is reflected in the dataset's design.

Key Results

The ES-JUST team achieved competitive rankings in the SemEval-2021 evaluation:

  • Sub-task 1A (Is-Humor): 26th rank with a 0.9564 F-score.
  • Sub-task 1B (Humor-Rating): 26th rank with a 0.5709 RMSE.
  • Sub-task 1C (Humor-Controversy): 25th rank with a 0.4888 F-score.
  • Sub-task 2A (Offense-Rating): 9th rank with a 0.4467 RMSE.

RoBERTa-Large was the best-performing model for most sub-tasks, demonstrating its superiority. The paper also includes confusion matrices for each sub-task, providing a detailed look at the model's performance in correctly classifying and misclassifying instances.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Dataset Bias: The paper notes that the training dataset for sub-task 1A (Is-Humor) is slightly biased towards the "humorous" label (label 1). This bias can affect the model's ability to accurately predict instances of non-humor.
  • Handling Subjectivity: The inherent subjectivity of humor and offense remains a challenge. The paper mentions that the model might struggle with cases where annotator agreement was low.
  • Future Work: The team plans to conduct more experiments and explore ensemble techniques to further enhance results. They also aim to address biased labels by incorporating more data or employing specific techniques to mitigate bias.

Summary updated

BibTeX
@inproceedings{bashabsheh2021es,
  author = {Emran Al Bashabsheh and Sanaa Abu Alasal},
  title = {{ES-JUST at SemEval-2021 Task 7: Detecting and Rating Humor and Offensive Text Using Deep Learning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure