TL;DR
This paper presents an approach for detecting and rating humor and offense in text by combining powerful transformer models with handcrafted lexical and "hurtlex" features. They demonstrate that task-adaptive pretraining significantly boosts performance, and that a combination of features yields the best results for humor detection and rating.
What Problem Does This Paper Address?
Humor is a complex aspect of human communication, and distinguishing it from offensive content, especially online, is a significant challenge. The rise of social media has amplified the need for automated systems that can accurately identify and categorize humorous and offensive text. This paper tackles the SemEval-2021 Task 7, "HaHackathon," which aims to develop models capable of detecting humor, rating its intensity, detecting controversy, and rating offense. The core challenge lies in building robust models that can handle the nuances of language, sarcasm, and cultural context inherent in humor and offense.
Methods
The IIITH team's approach leverages state-of-the-art transformer architectures, specifically RoBERTa-base and ELECTRA-base-discriminator, for their powerful language understanding capabilities. They explore two main avenues for improvement:
- Task Adaptive Pretraining (TAPT): Instead of directly fine-tuning pre-trained models, they further pre-train them on the specific task's unlabeled data using a masked language modeling objective. This "task adaptive pretraining" helps the models better understand the domain-specific language and nuances relevant to humor and offense detection.
- Feature Engineering: They augment the transformer embeddings with two types of handcrafted features:
- Lexical Features: These include basic text statistics like the count of letters, punctuation, uppercase letters, numbers, and the presence of named entities (identified using AllenNLP), interrogative words, and personal pronouns.
- Hurtlex Features: This leverages the HurtLex lexicon, a multilingual resource containing words categorized as offensive, aggressive, or hateful. This feature set directly targets the detection of potentially offensive language.
The final model concatenates the transformer embeddings with these handcrafted features before feeding them into a classification or regression head.
Datasets
The experiments were conducted on the dataset provided for SemEval-2021 Task 7: HaHackathon. This dataset consists of 10,000 sentences, with 8,000 for training, 1,000 for validation, and 1,000 for testing. Each entry includes labels for humor detection, humor rating, humor controversy, and offense rating.
Key Insights and Ideas
- TAPT is Crucial: Task adaptive pretraining significantly improves the performance of transformer models for humor and offense detection tasks. This highlights the importance of domain-specific adaptation for these complex language understanding challenges.
- Feature Fusion Matters: Combining transformer embeddings with handcrafted lexical and hurtlex features generally leads to better performance than using transformers alone. This suggests that while transformers excel at capturing contextual meaning, explicit linguistic cues can still provide valuable complementary information.
- Feature Roles Vary: The impact of lexical and hurtlex features differs across tasks. For humor classification and regression, the combination of both features with transformers yielded the best results. However, for offensive regression, hurtlex features were beneficial, while lexical features actually degraded performance, possibly due to their focus on humor-related cues. For humor controversy detection, excluding lexical and hurtlex features and relying solely on the transformer model proved most effective, suggesting that textual features alone were sufficient for this specific task.
Key Results
The IIITH approach achieved strong results on the SemEval-2021 Task 7:
- Humor Classification (Task 1a): An F1 score of 0.959 was achieved with the combined transformer, lexical, and hurtlex features.
- Humor Regression (Task 1b): An RMSE of 0.541 was obtained with the same feature combination.
- Offensive Regression (Task 2): An RMSE of 0.488 was achieved using the combined features.
- Humor Controversy (Task 1c): The best performance for this task was achieved using the ELECTRA-base-discriminator model without additional lexical or hurtlex features, yielding an F1 score of 0.375.
On the test split, their models achieved ranks of 14th for humor classification, 5th for humor regression, 6th for humor controversy, and 23rd for offensive regression.
Limitations and Future Work
The authors acknowledge that their experiments were conducted with default hyperparameters due to resource constraints. They suggest that hyperparameter optimization through cross-validation could further improve model performance. Additionally, for regression tasks, they propose exploring ensembles of the best-performing models to achieve even better predictions.
Summary updated