CSECU-DSG at SemEval-2021 Task 7: Detecting and Rating Humor and Offense Employing Transformers

Afrin Sultana, Nabila Ayman, Abu Nowshed ChyACL 2021

TL;DR

This paper tackles the challenge of automatically identifying and rating humorous and offensive online content, a complex task due to the subjective nature of humor and offense. The authors propose a novel framework that combines embeddings from multiple state-of-the-art transformer models (GPT-2, BERT, RoBERTa) using a stacked embedding approach, achieving competitive results in classification tasks.

What Problem Does This Paper Address?

The proliferation of user-generated content on online platforms has made it increasingly important to understand and process this data. Humor, in particular, presents a significant challenge for Natural Language Processing (NLP) due to its reliance on cultural context, multiple word senses, and pragmatic understanding. Furthermore, humorous content can sometimes be offensive, making it crucial to detect and rate both aspects. SemEval-2021 Task 7, "HaHackathon," was introduced to address this by focusing on detecting and rating humorous and offensive texts, encompassing tasks like binary humor detection, humor rating, humor controversy prediction, and offense rating. This paper details the CSECU-DSG team's approach to this challenging task.

Methods

The core of the proposed method lies in leveraging the power of transformer-based language models. The authors employ a stacked embedding scheme to combine features extracted from three pre-trained transformer models: GPT-2 medium, BERT-base (uncased), and RoBERTa-base. For each text, they extract a 1024-dimensional feature vector from GPT-2 medium, and 768-dimensional feature vectors from BERT-base and RoBERTa-base. These vectors are then concatenated to form a unified 2560-dimensional feature vector. This stacked embedding is then fed into a simple feed-forward linear architecture for classification and regression tasks. Additionally, they conduct experiments using individual fine-tuned BERT and RoBERTa models for comparison. For classification tasks (humor detection and controversy prediction), they use a linear classifier. For regression tasks (humor rating and offense rating), they utilize fine-tuned BERT and RoBERTa models.

Datasets

The study utilizes the benchmark dataset provided by the SemEval-2021 Task 7 organizers. This dataset consists of:

  • Training set: 8000 texts
  • Development set: 1000 texts (used for hyperparameter tuning)
  • Test set: 1000 texts (used for final evaluation)

Each entry in the dataset includes a unique identifier, the text itself, and labels for is_humor, humor_rating, humor_controversy, and offense_rating. Notably, for non-humorous texts, humor_rating and humor_controversy are marked as empty, which the authors address by converting these to a label of 0.

Key Insights and Ideas

  • Ensemble of Embeddings: Combining embeddings from multiple transformer models (GPT-2, BERT, RoBERTa) through stacking proves effective for classification tasks, capturing a broader range of linguistic nuances than individual models.
  • Transformer Dominance: The paper reinforces the state-of-the-art performance of transformer models in various NLP tasks, including humor and offense detection.
  • Classification vs. Regression Challenges: While the stacked embedding approach achieved competitive results in binary classification tasks (Subtask 1a and 1c), the system lagged behind in regression tasks (Subtask 1b and 2a). This highlights the inherent difficulty in accurately predicting nuanced scores for subjective qualities like humor and offense.
  • Data Scarcity for Regression: The limited number of perfectly humor-rated sentences in the training data (4932 out of 8000 total) is identified as a significant constraint for training robust regression models.

Key Results

The CSECU-DSG system achieved competitive performance in the SemEval-2021 Task 7.

  • For Subtask 1a (Humor Detection), their stacked embedding approach achieved an accuracy of 0.9380 and an F1-score of 0.9496, ranking 34th among participants.
  • For Subtask 1c (Humor Controversy), they reported an accuracy of 0.5366 and an F1-score of 0.4423, also ranking 34th.
  • In the regression tasks, their performance was less competitive. For Subtask 1b (Humor Rating), they achieved an RMSE of 0.6803 (ranking 41st), and for Subtask 2a (Offense Rating), their RMSE was 0.5395 (ranking 34th).

The paper notes that for classification tasks, the stacked embedding method was used, while for Subtask 1c, BERT's large model was employed. For regression tasks, both BERT's large and RoBERTa's base models were used.

Limitations and Future Work

The authors acknowledge several limitations:

  • Regression Performance: The system's performance in regression tasks (humor and offense rating) was not as strong as in classification tasks. This is attributed to the limited amount of perfectly humor-rated data available for training.
  • Intermediate Layer Exploration: They did not explore methods to identify and utilize the most effective intermediate layers of the transformer models for feature extraction.

Future work directions include:

  • Topic Information Integration: Incorporating topic information directly into transformer-based models to enhance their understanding of context.
  • Optimized Parameter Tuning for Regression: Developing more efficient methods for tuning parameters specifically for regression tasks to improve performance.

Summary updated

BibTeX
@inproceedings{sultana2021csecu,
  author = {Afrin Sultana and Nabila Ayman and Abu Nowshed Chy},
  title = {{CSECU-DSG at SemEval-2021 Task 7: Detecting and Rating Humor and Offense Employing Transformers}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure