TL;DR
This paper presents HumorHunter, a system for recognizing humor and offense in text, which leverages DeBERTa's disentangled attention mechanism. By effectively modeling the interplay between word content and relative position, their approach achieved strong results on the SemEval-2021 Task 7, particularly excelling in offense detection.
What Problem Does This Paper Address?
Humor is a complex and subjective human trait, making it challenging for machines to understand and generate. This paper tackles the problem of automatically detecting and rating humor and offense in short English texts, a task that has significant implications for human-computer interaction and content moderation. The SemEval-2021 Task 7, "HaHackathon," specifically aimed to address this by requiring systems to predict:
- Whether a text is humorous (binary classification).
- The average humor rating (regression).
- Whether the humor rating is controversial (binary classification).
- How offensive the text is (regression).
The inherent subjectivity of humor and offense, coupled with the nuances of language, makes this a difficult problem.
Methods
The core of the HumorHunter system is the DeBERTa architecture, which incorporates a disentangled attention mechanism. Unlike traditional Transformer models that rely on absolute positional embeddings, DeBERTa calculates attention weights by considering both the content vectors of words and their relative position vectors. This allows the model to better capture the relationships between words in a sequence, which is crucial for understanding the subtle cues that signal humor or offense.
The authors fine-tuned the pre-trained DeBERTa model on all four subtasks of the SemEval challenge. They experimented with both base and large versions of DeBERTa, finding that the larger model generally yielded better performance. For classification tasks (subtasks 1a and 1c), they used a softmax output layer with cross-entropy loss. For regression tasks (subtasks 1b and 2a), they employed mean squared error as the loss function.
Datasets
The SemEval 2021 Task 7 provided three datasets:
- Training set: 8,000 examples.
- Validation set: 1,000 examples.
- Final test set: 1,000 examples.
Each example in these datasets consists of a piece of English text annotated with four features corresponding to the four subtasks: is_humor, humor_rating (0-5), humor_controversy, and offense_rating (0-5). The paper also includes visualizations of word clouds for offensive and non-offensive examples, highlighting common terms associated with each.
Key Insights and Ideas
- Disentangled Attention is Key: The paper strongly suggests that disentangled attention, which separates content and relative position information, is a significant improvement over standard absolute positional embeddings for tasks involving nuanced language understanding like humor and offense.
- Transformer Dominance: Transformer-based models, particularly larger ones, consistently outperform traditional NLP methods (like Bag-of-Words or Bi-LSTMs) on these tasks, underscoring the power of pre-trained language models.
- Humor Controversy is Tricky: The authors note that all models struggled with the humor controversy subtask, attributing this to the highly subjective nature of the task, which is difficult even for humans to agree on.
- False Positives and Questions: A qualitative analysis of false positives in humor detection revealed that the system sometimes incorrectly flagged questions as humorous, likely due to the training data containing many humorous examples that were structured as questions followed by punchlines.
Key Results
The HumorHunter system, particularly the large DeBERTa version, achieved competitive results:
- Subtask 1a (Humor Detection): F-score of 0.9480.
- Subtask 1b (Humor Rating): RMSE of 0.5510.
- Subtask 1c (Humor Controversy): F-score of 0.4764.
- Subtask 2a (Offense Detection): RMSE of 0.4230, achieving rank 3 on the leaderboard.
The paper also shows that larger BERT-like models generally outperform their base counterparts, and Transformer models significantly outperform traditional methods.
Limitations and Future Work
- Subjectivity of Humor Controversy: The authors acknowledge the difficulty of the humor controversy subtask due to its inherent subjectivity.
- False Positives on Questions: The system exhibited a tendency to misclassify questions as humorous, suggesting a need for more robust handling of question-based humor.
For future work, the authors propose:
- Combining Human-Centric Features: Integrating features derived from human understanding of humor with the current neural architecture.
- Developing Hybrid Models: Creating models that blend different approaches for potentially better performance.
- Expanding Datasets: Augmenting the existing datasets with more jokes from diverse sources like Reddit forums to improve generalization.
Summary updated