FII FUNNY at SemEval-2021 Task 7: HaHackathon: Detecting and rating Humor and Offense

Mihai Samson, Daniela GifuACL 2021

TL;DR

This paper tackles the challenging task of automatically detecting and rating humor and offense in social media text, a notoriously subjective area. The authors explore various deep learning architectures, finding that fine-tuning a pre-trained BERT model yields the most promising results for humor detection, highlighting the power of transfer learning for this complex NLP problem.

What Problem Does This Paper Address?

Social media is awash with nuanced language, including humor, irony, and satire, which poses a significant challenge for Natural Language Processing (NLP). Accurately understanding and classifying these figurative expressions is crucial for applications ranging from content moderation to sentiment analysis. The SemEval-2021 Task 7, "HaHackathon: Detecting and Rating Humor and Offense," specifically aimed to address this by requiring participants to not only detect humor but also rate its intensity and offensiveness. This task is particularly difficult due to the inherent subjectivity of humor, making it hard even for humans to agree on classifications, let alone for AI models.

Methods

The authors experimented with several deep learning approaches. For models trained from scratch, they employed:

  • Convolutional Neural Networks (CNNs): Simple sequential models with 1D convolutions and max pooling layers. They found that even basic CNNs tended to overfit the data.
  • Bidirectional LSTM and GRU cells: These recurrent architectures were used to capture the sequential nature of language. Similar to CNNs, stacking multiple cells led to overfitting.
  • Transformer blocks: They implemented the encoder part of the "Attention is All You Need" architecture, followed by global average pooling.

Crucially, they also explored fine-tuning a pre-trained Transformer model (BERT). This approach leverages the rich linguistic representations learned by BERT on massive datasets and adapts them to the specific task of humor and offense detection using the provided dataset. For text preprocessing, they used tokenization and mapped words to GloVe embeddings for models trained from scratch, while BERT utilized its own specific tokenization and embeddings.

Datasets

The primary dataset used in this study is from the SemEval-2021 Task 7: HaHackathon, Detecting and Rating Humor and Offense. This dataset contains 8000 short texts, each with four labels:

  • A binary label indicating if the text is humorous.
  • A score from 0 to 5 representing the average humorous score.
  • A binary label indicating if the humor is controversial.
  • A score from 0 to 5 denoting if the text is offensive.

The authors note that for the "controversy" subtask, labels appeared to be randomly assigned, making it impossible for their models to achieve better than chance performance. They used the entire dataset for the binary humor detection and offense rating tasks, and only humorous texts for the humor rating and controversy tasks.

Key Insights and Ideas

  • Transfer learning is key: Fine-tuning a pre-trained BERT model significantly outperformed models trained from scratch, underscoring the advantage of leveraging large-scale pre-trained language representations for complex, subjective tasks like humor detection.
  • Subjectivity is a major hurdle: The "controversy" subtask proved exceptionally difficult, with annotator disagreement making it nearly impossible for DL models to achieve meaningful results. This highlights the inherent challenges in quantifying subjective human perception.
  • Overfitting is a persistent problem: Even with regularization techniques, the models tended to overfit the training data, a common issue when working with smaller, specialized datasets. This suggests that data augmentation or larger, more diverse datasets might be necessary for further improvements.
  • Simpler models can overfit quickly: The authors observed that even basic CNN architectures could easily overfit the data, reinforcing the need for careful model selection and regularization.

Key Results

  • Humor Detection: The fine-tuned BERT model achieved the highest accuracy on the validation set at 95.4% for the humor detection task. This was a significant improvement over models trained from scratch.
  • Regression Tasks: For the "Average Humor" rating, the model achieved an RMSE of 0.5598, ranking 19th out of 50. For "Average Offensiveness," the RMSE was 0.4788, ranking 24th out of 48.
  • Official Competition Performance: The authors reported a lower accuracy of 92.2% for humor detection in the official competition due to an accidental mistake in submitting labels.
  • Controversy Task: Performance on the controversy task was around 50%, indicating performance no better than random chance.

Limitations and Future Work

  • Overfitting: The authors acknowledge that all their methods overfitted the training set, even with dropout and regularization. They suggest that these techniques only delayed overfitting rather than preventing it.
  • Small Dataset Size: The limited size of the dataset is a significant constraint. The authors suggest that data augmentation techniques, such as replacing words with synonyms, and using similar datasets could help increase the dataset size and improve performance.
  • Subjectivity of Humor: The difficulty in the "controversy" task highlights the challenge of capturing subjective human interpretations. Future work could explore methods to better model inter-annotator agreement or incorporate human-in-the-loop approaches.
  • Adapting Pre-trained Models: The success of fine-tuning BERT suggests that further exploration of various pre-trained models and their adaptation strategies for humor and offense detection is a promising avenue.

Summary updated

BibTeX
@inproceedings{samson2021fii,
  author = {Mihai Samson and Daniela Gifu},
  title = {{FII FUNNY at SemEval-2021 Task 7: HaHackathon: Detecting and rating Humor and Offense}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure