DuluthNLP at SemEval-2021 Task 7: Fine-Tuning RoBERTa Model for Humor Detection and Offense Rating

Samuel AkrahACL 2021

TL;DR

This paper presents a fine-tuned RoBERTa model for the SemEval-2021 Task 7, which aims to detect and rate humor and offense in text. By systematically optimizing hyperparameters like learning rate, batch size, and epochs using Bayesian optimization, the authors achieved strong performance on humor detection and offense rating tasks, demonstrating the effectiveness of fine-tuning large language models for nuanced linguistic understanding.

What Problem Does This Paper Address?

Humor detection and offense rating are complex natural language processing (NLP) tasks that require understanding context, culture, and subtle linguistic cues like irony and sarcasm. While previous SemEval tasks have focused on either humor detection or offense detection separately, SemEval-2021 Task 7 uniquely combines these challenges, requiring models to not only identify humor but also rate its offensiveness. This presents a significant challenge due to the subjective nature of both humor and offense, and the potential for overlap where humorous content can also be offensive.

Methods

The core of the DuluthNLP submission is a RoBERTa model, a powerful transformer-based language model known for its state-of-the-art performance on various NLP tasks. The authors utilized the HuggingFace Transformers library for implementing RoBERTa. Their approach involved two main stages of hyperparameter tuning:

  1. Manual Tuning: Initially, a manual exploration of hyperparameters (learning rate, batch size, epochs) was performed on Task 1a (humor classification) to establish baseline settings.
  2. Bayesian Optimization: For a more systematic and efficient search, Bayesian optimization was employed. This technique uses a probabilistic model to guide the search for optimal hyperparameters, making it more efficient than grid or random search, especially for expensive functions (like training a large model). The Bayesian optimization was applied to fine-tune learning rates across different layers of the RoBERTa model, as well as batch size and epochs, for each subtask.

Datasets

The paper utilizes the datasets provided for SemEval-2021 Task 7. These datasets consist of texts annotated for humor (binary classification: humorous/not humorous) and humor rating (scale of 1-5). Additionally, texts are labeled for offense rating (scale of 0-5) and controversy. The data splits for training and evaluation were 80% and 20%, respectively. The annotators for these datasets were diverse in age, gender, political views, and income levels to capture a range of perceptions.

Key Insights and Ideas

  • RoBERTa's Efficacy: The paper reinforces the effectiveness of pre-trained transformer models like RoBERTa for complex NLP tasks requiring contextual understanding, even with minimal task-specific architectural changes.
  • Hyperparameter Sensitivity: Fine-tuning hyperparameters, particularly learning rate, batch size, and epochs, significantly impacts performance on humor detection and offense rating tasks.
  • Bayesian Optimization's Value: Bayesian optimization proves to be a valuable tool for efficiently finding optimal hyperparameter configurations, leading to substantial performance gains compared to manual tuning or simpler search strategies.
  • Humor vs. Offense Interplay: The results suggest a strong correlation between humor and offense detection. The model's tendency to misclassify offensive content as humorous, and vice-versa, highlights the inherent overlap and the challenge of disentangling these two aspects.
  • Offense Rating as a Stronger Signal: The paper hints that the model might be better at offense rating than humor rating, as improved performance on Task 1a (humor detection) might be driven by its ability to detect offensive content within humorous texts.

Key Results

The DuluthNLP submission achieved strong results on SemEval-2021 Task 7:

  • Task 1a (Humor Detection): An F1-score of 0.939 was achieved during official evaluation, which improved to 0.957 after post-evaluation hyperparameter sweeps.
  • Task 1b (Humor Rating): The official RMSE score was 0.646, improving to 0.5802 post-evaluation.
  • Task 2a (Offense Rating): The official RMSE score was 0.506, improving to 0.469 post-evaluation.

The post-evaluation results, particularly the gains in RMSE for Task 1b and Task 2a, demonstrate the significant impact of the Bayesian optimization process.

Limitations and Future Work

  • Limited Initial Tuning Scope: A key limitation acknowledged is that the initial hyperparameter tuning was performed only on Task 1a (binary classification), and these settings were then applied to the regression tasks (Task 1b and Task 2a). This might explain the initially subpar performance on the regression tasks.
  • Model Architecture: The RoBERTa model was used "as is" without modifications to its classification layer or pre-trained layers. Future work could explore how modifying these layers might further impact performance.
  • Environmental and Bias Concerns: The paper touches upon the significant energy cost of training large language models and the potential for embeddings to amplify societal biases. Future work could focus on minimizing bias and energy consumption.
  • English Language Bias: The annotators were limited to English speakers, meaning the system's ability to detect humor is largely reflective of views inherent in the English language. Expanding to multilingual datasets would be a valuable future direction.
  • Offensive Content Misclassification: The model's propensity to classify offensive content as humorous is a notable limitation, suggesting that applications might be more likely to deem offensive content as humorous. Further research is needed to address this.

Summary updated

BibTeX
@inproceedings{akrah2021duluthnlp,
  author = {Samuel Akrah},
  title = {{DuluthNLP at SemEval-2021 Task 7: Fine-Tuning RoBERTa Model for Humor Detection and Offense Rating}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure