TL;DR
This paper presents a system for detecting and rating humor and offense in text, tackling the inherent subjectivity and context-dependency of these phenomena. The authors leverage pre-trained language models like BERT, ALBERT, and RoBERTa, combined with CNNs and regression techniques, to achieve strong performance on the SemEval-2021 Task 7.
What Problem Does This Paper Address?
Humor and offense are complex linguistic phenomena that are notoriously difficult for machines to understand. Unlike factual statements, humor is highly subjective, influenced by cultural context, personal background, and even the specific situation. What one person finds funny, another might not, and humor can sometimes border on or directly involve offensive content. This subjectivity makes it challenging to build reliable automated systems for detecting and rating humor and offense. SemEval-2021 Task 7 aimed to address this by proposing four sub-tasks: predicting if a text is humorous, rating its degree of humor, determining if humor is controversial, and rating its offensiveness.
Methods
The authors propose two main methods to tackle the four sub-tasks.
Method A (for Sub-task 1a and 1c - Binary Classification): This method utilizes a BERT-base pre-trained model as the foundation. The contextualized embeddings from BERT's last four hidden layers are fed into a Convolutional Neural Network (CNN) architecture. This CNN then extracts features, which are passed to a dense layer for classification. The authors experimented with various pre-trained models, including BERT, ALBERT, and RoBERTa, and found that combining BERT with CNNs yielded good results.
Method B (for Sub-task 1b and 2 - Regression): For tasks requiring a continuous output (degree of humor and offense), the authors employed regression techniques. They leveraged pre-trained models like BERT, ALBERT, and RoBERTa for word embeddings. RoBERTa, in particular, showed superior performance in regression tasks compared to BERT. The trained models were then used to predict humor and offense ratings on a scale of 0 to 5.
Datasets
The data for all four sub-tasks was provided by SemEval (Meaney et al., 2021). The authors employed a stratified 5-fold cross-validation approach on the provided training and test sets. The training set contained 8000 data points, and the test set had 1000. For their internal development, they split the training set into 85% for training and 15% for a development set.
Key Insights and Ideas
- Pre-trained models are crucial: The paper strongly emphasizes the effectiveness of pre-trained language models like BERT, ALBERT, and RoBERTa for capturing nuanced linguistic features essential for understanding humor and offense.
- CNNs complement BERT for classification: For binary classification tasks (humor detection and controversy), combining BERT's contextual embeddings with CNNs proved to be a powerful approach for feature extraction.
- RoBERTa excels in regression: When dealing with regression tasks (rating humor degree and offense), RoBERTa demonstrated superior performance over BERT, likely due to its more robust pre-training strategy.
- Subjectivity is a major hurdle: The authors acknowledge the inherent subjectivity of humor and offense, noting that different people have varying interpretations, which poses a significant challenge for automated systems.
Key Results
The paper reports achieving good results across all four sub-tasks.
- Sub-task 1a (Humor Detection): The best F1-score achieved was 0.9206.
- Sub-task 1c (Humor Controversy): The best F1-score achieved was 0.6744.
- Sub-task 1b (Humor Rating): The best Root Mean Square Error (RMSE) was 0.6510.
- Sub-task 2 (Offense Rating): The best RMSE was 0.5588.
The authors note that their results on the leaderboard were slightly different from the official evaluation, but still competitive. They found that CNNs had a noticeable effect on text classification, even with a single layer.
Limitations and Future Work
- Limited CNN depth: The authors mention that only one layer of CNN was added to the classifier, suggesting that exploring deeper CNN architectures could potentially improve performance.
- Future integration and distillation: The authors plan to integrate and distill their models, which could lead to more efficient and potentially more accurate systems.
- External knowledge: They also consider introducing external knowledge to better model headlines and enhance humor recognition performance.
Summary updated