TL;DR
This paper presents a fine-tuning approach using various pre-trained language models (PLMs) for the SemEval-2021 Task 7: HaHackathon, which focuses on detecting and rating humor and offense. The authors found that averaging predictions from multiple PLMs significantly improved performance on regression tasks, demonstrating the effectiveness of ensemble methods for this challenging NLP problem.
What Problem Does This Paper Address?
Humor is a complex and subjective aspect of human communication, making it difficult for machines to understand and analyze. Jokes can be funny to some but offensive to others, and their intensity varies greatly. SemEval-2021 Task 7: HaHackathon aimed to tackle these challenges by providing a dataset for detecting humor, rating its intensity, and predicting whether a joke might be controversial or offensive. This task is crucial for developing more nuanced and socially aware NLP systems, especially in applications like content moderation and personalized content delivery.
Methods
The core of the system relies on fine-tuning pre-trained language models (PLMs). The authors experimented with several state-of-the-art PLMs, including BERT, ELECTRA, RoBERTa, XLNet, MPNet, and ALBERT. For each PLM, a simple architecture was employed: a single-neuron dense layer was added on top of the PLM's features, with a sigmoid activation for classification tasks and no activation for regression tasks.
A key innovation was the averaging technique for regression tasks. For SubTask 1-b (rating humor intensity), they averaged the predictions of all fine-tuned models. For SubTask 2 (rating offense), they employed a more cautious averaging strategy: predictions were averaged only if all models predicted a non-zero offense rating; otherwise, zero was taken as the final prediction. This was motivated by the observation that a significant portion of the training data had zero offense ratings.
Datasets
The experiments were conducted on the dataset provided for SemEval-2021 Task 7: HaHackathon. This dataset includes:
- SubTask 1-a: Binary classification of humorous vs. non-humorous text.
- SubTask 1-b: Regression task to quantify humor intensity (0-5).
- SubTask 1-c: Binary classification of controversial vs. non-controversial humorous text.
- SubTask 2: Regression task to quantify offense intensity (0-5).
The paper provides statistics for these tasks, noting a slight class imbalance in SubTask 1-a and a significant number of zero offense ratings in SubTask 2.
Key Insights and Ideas
- Fine-tuning PLMs is effective: The study confirms that fine-tuning powerful pre-trained language models is a strong baseline for humor and offense detection tasks.
- Ensembling PLMs boosts regression performance: Averaging predictions from multiple PLMs significantly improved results for the regression tasks (SubTask 1-b and SubTask 2), highlighting the benefit of ensemble methods.
- Careful averaging for offense detection: The specific averaging strategy for SubTask 2, which accounted for the prevalence of zero offense ratings, proved crucial for achieving better performance.
- Subjectivity of humor is a challenge: The results for SubTask 1-c (controversial text detection) show that while models can identify non-controversial text well, predicting controversial humor remains difficult, underscoring the inherent subjectivity.
Key Results
The system achieved competitive results across the subtasks:
- SubTask 1-b: Ranked 4th.
- SubTask 1-c: Ranked 8th.
- SubTask 2: Ranked 12th.
- SubTask 1-a: Performed well.
Specifically, the PLM averaging technique for SubTask 2 provided a significant improvement in RMSE compared to individual models. For SubTask 1-c, the model performed well in identifying non-controversial text but struggled with precision for controversial text due to high false positives.
Limitations and Future Work
- Computational Cost: The authors acknowledge that PLMs are computationally expensive, and fine-tuning them can be resource-intensive.
- Model Selection: Choosing the right PLM for a specific downstream task can be challenging due to varying performance.
- Future Work: The authors suggest exploring the effect of pre-training PLMs with additional task-specific data and investigating how this might further improve performance on downstream tasks. They also plan to continue working on the challenges of humor detection.
Summary updated