RoMa at SemEval-2021 Task 7: A Transformer-based Approach for Detecting and Rating Humor and Offense

Roberto Labadie, Mariano Jason Rodriguez, Reynier Ortega, Paolo RossoACL 2021

TL;DR

This paper presents RoMa's approach to SemEval-2021 Task 7, focusing on detecting and rating humor and offense in English tweets. They propose two novel Transformer-based architectures, one using a Siamese Neural Network for humor detection and another employing a multi-view linear regression model for offense rating, achieving encouraging results.

What Problem Does This Paper Address?

The paper tackles the complex challenge of computationally distinguishing between humor and offense in social media text, particularly tweets. This is a difficult task because offensive content is often masked as humor, and understanding the true intent requires nuanced linguistic and contextual understanding. The authors highlight the growing prevalence of harmful content disguised as jokes, making accurate detection crucial for social media platforms and online interactions. They aim to address this by developing models that can not only identify humor and offense but also rate their intensity.

Methods

The RoMa team developed two distinct Transformer-based architectures:

  • Siamese Neural Network (SNN) for Humor Detection: This approach uses a Siamese Neural Network to learn a latent space where humorous and non-humorous messages are distinguished. They employ a graph-based clustering method to create "prototypes" for each class, which are then used for training and classifying new messages. The SNN learns to measure the distance between message representations, aiming to place similar messages closer and dissimilar ones further apart.
  • Multi-view Linear Regression for Offense Rating: This architecture combines deep representations from fine-tuned Transformer models (RoBERTa, BERTweet, XLNet) with affective features extracted by a BiLSTM. These fused representations are then fed into a linear regression model to predict offense scores. They also explored multitask learning (MTL) strategies, combining offense rating with irony detection, which proved beneficial.

Both architectures leverage pre-trained Transformer models (RoBERTa, XLNet, BERTweet) for initial feature extraction. They experimented with different strategies for selecting the most informative output from the Transformer sequence, including the CLS token, normalized sum, and last token.

Datasets

The primary dataset used is the HaHackathon development dataset, which contains approximately 8000 tweets for training and 1000 for testing, manually annotated for humor, offense, controversy, and humor/offense ratings. The authors also utilized datasets from previous SemEval tasks (Irony Detection Task at SemEval 2018, HatEval 2019, OffensEval 2019) for fine-tuning their Transformer encoders, particularly for multitask learning.

Key Insights and Ideas

  • The Masking of Offense in Humor: The paper confirms the observation that offensive messages are often disguised as humor, making their distinction challenging. This is evidenced by the overlap in scores and the use of humor devices to convey hateful content.
  • Prototype-based Classification: Using graph-based clustering to identify representative "prototypes" for humor and non-humor classes proved effective for the Siamese network, allowing it to learn a robust decision boundary.
  • Multitask Learning for Offense: Combining offense rating with irony detection through multitask learning significantly improved performance, suggesting that understanding indirect negative speech is crucial for accurate offense detection.
  • Affective Features Matter: The inclusion of affective features, captured by a BiLSTM, played a vital role in the linear regression model for offense rating, demonstrating that emotional cues are important indicators of offensiveness.
  • Fine-tuning Strategies are Crucial: The choice of fine-tuning strategy for the Transformer encoders, including the use of dynamic learning rates (ULMFiT) and multitask learning, had a significant impact on model performance.

Key Results

  • Humor Detection (Subtask 1a): The Siamese Neural Network approach achieved a high F1-Score/Accuracy of 0.9505/0.9370 on the test set.
  • Humor Rating (Subtask 1b): The Multiview-based Linear Regression model achieved a RMSE of 0.5905.
  • Controversial Humor Prediction (Subtask 1c): The BERTweet model, using MTL with all four subtasks, achieved an F1-Score/Accuracy of 0.4732/0.6197.
  • Offensiveness Rating (Subtask 2): The Multiview-based Linear Regression model, fine-tuned with MTL and irony detection, achieved a RMSE of 0.4532.
  • The paper notes that their systems achieved competitive results, ranking well within the overall competition.

Limitations and Future Work

  • Misclassification of Q&A Structures: In humor prediction, a significant portion of misclassifications involved tweets with a Question/Argumentation (Q&A) structure, suggesting that this common humorous format can sometimes lead to false positives.
  • Bias in Pre-training Data: For offense prediction, the model sometimes assigned higher offensiveness values to messages containing phrases targeting social groups, potentially due to biases in the pre-training data of the Transformer encoders.
  • Future Work: The authors plan to conduct an in-depth study on the harmfulness of humor on human stereotypes, further exploring the overlap between humor and offense. They also intend to investigate different clustering methods for building prototypes and their impact on Siamese network learning for humor prediction.

Summary updated

BibTeX
@inproceedings{labadie2021roma,
  author = {Roberto Labadie and Mariano Jason Rodriguez and Reynier Ortega and Paolo Rosso},
  title = {{RoMa at SemEval-2021 Task 7: A Transformer-based Approach for Detecting and Rating Humor and Offense}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure