Grenzlinie at SemEval-2021 Task 7: Detecting and Rating Humor and Offense

Renyuan Liu, Xiaobing ZhouACL 2021

TL;DR

This paper tackles the challenging task of automatically detecting and rating humor and offense in text, going beyond simple binary classification. The authors propose a novel "compare method" and leverage Bi-LSTM with pre-trained models like RoBERTa, achieving state-of-the-art results on SemEval-2021 Task 7, particularly in humor detection and rating.

What Problem Does This Paper Address?

Humor is a complex and nuanced aspect of human communication, making it difficult for machines to understand and replicate. While previous research has focused on generating humorous text or simple humor detection, this paper addresses the need for more sophisticated understanding: not only detecting humor but also rating its intensity and distinguishing it from offensive content. This is crucial for building more engaging and context-aware conversational AI systems. The SemEval-2021 Task 7, which this paper participates in, aims to advance this by introducing subtasks for humor detection, humor rating prediction, humor controversy detection, and offensive rating prediction.

Methods

The authors build upon the powerful RoBERTa pre-trained language model as their baseline. For classification tasks (humor detection and controversy detection), they incorporate a Bi-LSTM layer to extract richer sequential features and explore adversarial training (specifically FGM - Fast Gradient Method) to improve model robustness. They also experiment with FocalLoss to address potential data imbalance issues. For regression tasks (humor rating and offensive rating), Bi-LSTM is again employed. A key novel contribution is the "Compare Method," which extends traditional few-shot comparison techniques to regression. This method encodes pairs of sentences and their ratings, then uses the differences and sums of these ratings to train the model, aiming to better capture the nuances of humor intensity.

Datasets

The experiments were conducted on datasets for SemEval-2021 Task 7, which includes benchmarks for humor detection, humor rating, humor controversy detection, and offensive rating. The paper notes that for the humor scoring and controversy detection tasks, only humorous sentences are labeled with ratings and controversy labels, posing a challenge for handling non-humorous sentences.

Key Insights and Ideas

  • Beyond Binary: Simple binary classification for humor detection is insufficient; rating intensity and distinguishing offense are critical for a deeper understanding.
  • Bi-LSTM's Power: Integrating Bi-LSTM with pre-trained models like RoBERTa significantly enhances feature extraction, leading to improved performance across multiple tasks.
  • Adversarial Training Benefits: FGM-based adversarial training proves effective in improving performance on humor detection, rating, and offensive rating tasks, suggesting that robustness is key.
  • FocalLoss Limitations: FocalLoss, while designed for imbalanced datasets, did not perform as expected in this specific humor detection context, possibly due to the nature of the label distribution (more humorous than non-humorous, but not in the typical way FocalLoss is applied).
  • "Compare Method" Nuances: The proposed "Compare Method" shows promise, particularly for offensive rating prediction, but its effectiveness for humor rating was hindered by the prevalence of non-humorous sentences with a zero rating, which interfered with the comparison process.
  • Controversy Detection Challenge: The controversy detection task proved particularly difficult, with the baseline model outperforming more complex approaches. This is attributed to the challenge of defining and detecting controversy, especially when dealing with non-humorous content.

Key Results

The "Grenzlinie" team achieved impressive results on SemEval-2021 Task 7:

  • Humor Detection: 95.05% F1-score (using Bi-LSTM + FGM).
  • Humor Rating: 0.6143 RMSE (using Bi-LSTM).
  • Offensive Rating: 0.4761 RMSE (using the "Compare Method").
  • Humor Controversy Detection: 61.74% F1-score (baseline RoBERTa performed best).

Limitations and Future Work

The authors acknowledge that the controversy detection task remains a significant challenge, with the baseline model performing best. They attribute this to the difficulty in defining and detecting controversy, especially when non-humorous sentences are involved. Future work could focus on better methods to eliminate the influence of non-humorous sentences, exploring alternative pre-trained models, and potentially combining classification and regression models or using techniques like Bayesian inference or Conditional Random Fields (CRF) for processing BERT outputs. The "Compare Method" also requires further refinement to better handle zero-rated sentences in humor rating tasks.

Summary updated

BibTeX
@inproceedings{liu2021grenzlinie,
  author = {Renyuan Liu and Xiaobing Zhou},
  title = {{Grenzlinie at SemEval-2021 Task 7: Detecting and Rating Humor and Offense}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure