Gulu at SemEval-2021 Task 7: Detecting and Rating Humor and Offense

Maoqin YangACL 2021

TL;DR

This paper presents the "Gulu" team's approach to SemEval-2021 Task 7, focusing on detecting and rating humor and offense in English text. They leverage pre-trained transformer models like ALBERT and RoBERTa, combined with recurrent neural networks (BiGRU, BiLSTM) and attention mechanisms, to achieve strong performance across four subtasks.

What Problem Does This Paper Address?

Humor recognition is a notoriously difficult task in Natural Language Processing (NLP). It's not just about identifying a joke; it's about understanding the nuances of language, cultural context, and even the potential for humor to mask offense. This paper tackles SemEval-2021 Task 7, which aims to automatically detect humor and rate its offensiveness. The challenge lies in the subjective nature of humor and the need for models to grasp subtle linguistic cues, irony, and satire, all while considering the diverse backgrounds of annotators.

Methods

The Gulu team employed a multi-pronged strategy, utilizing state-of-the-art transformer models as their foundation.

  • Subtask 1a (Humor Detection): They used ALBERT, a "lite" version of BERT, as the base model. To enhance its classification capabilities, they integrated a BiGRU-Attention mechanism. This involves feeding ALBERT's output into a Bidirectional Gated Recurrent Unit (BiGRU) to capture sequential information, followed by an attention layer to weigh the importance of different parts of the text, and finally a softmax layer for classification.
  • Subtask 1c (Humor and Offense Classification): For this subtask, they opted for RoBERTa, an optimized version of BERT, and combined it with a BiLSTM-BiGRU architecture. This approach aims to capture bidirectional semantic dependencies more effectively.
  • Subtasks 1b and 2 (Humor and Offense Rating): These are regression tasks, where the goal is to predict a numerical rating. For these, they used the standard BERT model.

A key methodological choice was the use of 5-fold cross-validation to maximize the utilization of limited data and improve model robustness.

Datasets

The primary dataset used for this task was provided by the SemEval-2021 Task 7 organizers. This dataset was collected from annotators aged 18-70, representing diverse genders, political stances, and income levels. The dataset was divided into training, development, and testing sets. The paper notes that some data was missing for specific subtasks (e.g., if a sentence was not humorous in 1a, its corresponding humor rating data for 1b and 1c was absent). The authors addressed this by imputing missing values with 0.

Key Insights and Ideas

  • Transformer Powerhouse: The paper reinforces the dominance of transformer-based models (BERT, ALBERT, RoBERTa) in NLP tasks, demonstrating their effectiveness even in nuanced areas like humor detection.
  • Hybrid Architectures Shine: Combining transformers with recurrent neural networks (BiGRU, BiLSTM) and attention mechanisms proved crucial for capturing both semantic meaning and sequential dependencies, leading to improved performance.
  • Data Imbalance is a Hurdle: The authors explicitly point out that the low score in subtask 1c might be due to data imbalance and the dependency on subtask 1a, suggesting that careful handling of such issues is vital.
  • Subjectivity of Humor: The paper acknowledges the inherent difficulty in defining and computationally understanding humor due to its loose definition, reliance on context, and cultural background.

Key Results

The Gulu team achieved competitive results across the four subtasks:

  • Subtask 1a (Humor Detection): Weighted average F1 score of 0.9190.
  • Subtask 1b (Humor Rating): RMSE score of 0.7405.
  • Subtask 1c (Humor and Offense Classification): Weighted average F1 score of 0.5561.
  • Subtask 2 (Offense Rating): RMSE score of 0.5807.

The paper also presents comparative results for subtask 1c, showing that their RoBERTa+BiGRU+BiLSTM model achieved an F1-score of 0.88, outperforming the base RoBERTa model (0.80) and RoBERTa+BiGRU (0.86).

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Data Imbalance: The imbalance in the dataset for subtask 1c is identified as a significant factor affecting performance.
  • Subjectivity and Context: The inherent difficulty in creating a computational mechanism to understand humor due to its subjective and context-dependent nature is a major challenge.
  • Future Directions: The authors emphasize the need to design features that are interpretable, calculable, and easily implementable, aligning with humor theory. This suggests a move towards more explainable AI in this domain.

Summary updated

BibTeX
@inproceedings{yang2021gulu,
  author = {Maoqin Yang},
  title = {{Gulu at SemEval-2021 Task 7: Detecting and Rating Humor and Offense}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure