TL;DR
This paper presents a deep neural network system for SemEval-2021 Task 7, which aims to detect humor and offense in text, as well as predict their ratings. The system leverages fine-tuned BERT models with either linear layers or Bi-LSTM layers, achieving strong performance across all four subtasks and outperforming baseline models.
What Problem Does This Paper Address?
Detecting humor and offense in text is a challenging task due to its subjective nature, varying across individuals and demographics. SemEval-2021 Task 7 specifically addresses this by introducing four subtasks: binary classification of humor and offense, and regression for humor and offense ratings (on a scale of 0-5). This task is crucial for applications like chatbots and virtual assistants to better understand and respond to user sentiment.
Methods
The core of the system relies on fine-tuning pre-trained BERT models. Two main approaches are explored for processing BERT's output features:
- Linear Layers: Multiple linear layers (specifically, three layers with varying dimensions) are stacked on top of BERT's extracted features. Rectified Linear Unit (ReLU) activation and dropout are used for regularization. This approach is used for both classification and regression tasks.
- Bi-LSTM Layers: A Bidirectional Long Short-Term Memory (Bi-LSTM) layer is employed to capture sequential context from BERT's features, followed by a linear layer for the final prediction. This is also applied to both classification and regression.
For classification tasks (humor and offense detection), a softmax output layer is used. For regression tasks (humor and offense ratings), a mean squared error (MSE) loss function is employed. The system uses the Adam optimizer for training.
Datasets
The dataset for SemEval-2021 Task 7 was used. It consists of English texts with annotations from a diverse group of annotators (aged 18-70, varied genders, political stances, and income levels). The training set contains 8000 texts, with development and test sets each containing 1000 texts. The dataset includes labels for:
is_humor(binary: humorous or not)humor_rating(0-5 scale)is_offensive(binary: offensive or not)offense_rating(0-5 scale)
The paper also notes the humor_controversy label, which indicates if a text's humor rating had high variance among annotators.
Key Insights and Ideas
- BERT's Power: The paper reinforces the effectiveness of BERT as a powerful foundation model for NLP tasks, even when fine-tuned with relatively simple downstream architectures.
- Linear vs. Sequential: For classification tasks (humor and offense detection), a linear approach on top of BERT features yielded better results than a Bi-LSTM. This suggests that for these specific tasks, the rich contextual embeddings from BERT might be sufficient without explicit sequential modeling.
- Bi-LSTM for Ratings: Conversely, for the regression tasks (predicting humor and offense ratings), the Bi-LSTM approach outperformed the linear layer approaches. This indicates that capturing sequential dependencies is more beneficial for nuanced rating predictions.
- Task-Specific Fine-tuning: The system demonstrates that tailoring the fine-tuning architecture (linear vs. Bi-LSTM) to the specific task (classification vs. regression) can lead to improved performance.
Key Results
The system achieved strong results across all four subtasks:
- Humor Classification (Subtask 1): F1 score of 0.92 (linear approach) and 0.9177 (Bi-LSTM approach), outperforming the Naive Bayes baseline (0.8840).
- Humor Rating (Subtask 2): RMSE of 0.57 (3 linear layers) and 0.56 (Bi-LSTM approach), significantly lower than the SVR baseline (0.8609).
- Offense Classification (Subtask 3): F1 score of 0.56 (linear approach) and 0.5455 (Bi-LSTM approach), outperforming the Naive Bayes baseline (0.4624).
- Offense Rating (Subtask 4): RMSE of 0.58 (3 linear layers) and 0.58 (Bi-LSTM approach), significantly lower than the SVR baseline (0.6415).
The paper also mentions their ranking in the competition, noting that while they performed well, there's still room for improvement compared to the top-ranked teams.
Limitations and Future Work
- Sarcasm and Context: The authors acknowledge that their model struggles with detecting underlying sarcasm, especially in sentences where context is explored for the first time. This is highlighted with an example where the model misclassified a text as not humorous when it was intended to be.
- Further Exploration: While not explicitly detailed as future work, the paper's exploration of different fine-tuning strategies (linear vs. Bi-LSTM) suggests that further architectural experimentation could yield even better results. Investigating more sophisticated methods for handling sarcasm and nuanced context could be a promising direction.
Summary updated