TL;DR
This paper tackles the complex task of automatically detecting and rating humor and offense in text, a challenge complicated by subjectivity and cultural nuances. The authors leverage BERT-based models, exploring various architectures and ensemble techniques, achieving strong results on the SemEval-2021 Task 7 benchmark. Their work highlights the power of pre-trained language models and ensemble methods for nuanced linguistic understanding.
What Problem Does This Paper Address?
Humor is notoriously difficult to define and detect, as its perception is highly subjective and influenced by individual background, culture, and context. What one person finds funny, another might find offensive or simply not amusing. This subjectivity extends to rating the intensity of humor and identifying when humor might be controversial (i.e., when annotators disagree significantly on its rating). Furthermore, distinguishing between humor and offense, and rating them independently, presents a unique challenge. SemEval-2021 Task 7 aimed to address these issues by providing a benchmark for automatically detecting humor, rating its intensity, identifying controversial humor, and predicting general offensiveness.
Methods
The core of the authors' approach relies on BERT (Bidirectional Encoder Representations from Transformers), a powerful pre-trained language model. They experimented with several BERT-based architectures:
- BERT-Large fine-tuning (BERT-L): Directly fine-tuning the BERT-Large model for classification tasks.
- BERT-Large with a Feed-Forward Network (BERT-FFN): Adding a fully connected layer on top of BERT's output for regression tasks.
- BERT-Large Textual Entailment (BERT-ENT): Adapting BERT for textual entailment by rephrasing the task to determine if a statement like "It is humorous" can be inferred from the input text.
- BERT-Large with a Convolutional Neural Network (BERT-CNN): Using BERT to generate sentence embeddings and then applying a 1D CNN to these embeddings.
- BERT-Large based Ensemble (BERT-ENS): This was their most successful approach, employing bagging. They trained multiple BERT-Large models (specifically, the BERT-FFN architecture) on different subsets of the training data and combined their predictions through voting or averaging. This ensemble strategy helps to reduce variance and improve robustness.
For regression tasks (humor rating and offensiveness score), they used Mean Squared Error (MSE) as the loss function. For classification tasks (humor detection and controversy detection), they used Binary Cross-Entropy. They also explored meta-learning for controversy detection by tuning softmax score thresholds to account for class imbalance.
Datasets
The primary dataset used was provided by the SemEval-2021 Task 7 organizers. This dataset consisted of:
- 8000 English text sentences for training.
- 1000 English text sentences for development.
- 1000 English text sentences for evaluation.
The dataset contained labels for all sub-tasks: humor detection, humor rating, controversy detection, and offense prediction. No external datasets were used for training.
Key Insights and Ideas
- BERT's Dominance: The paper strongly reinforces the idea that pre-trained language models like BERT are highly effective for nuanced linguistic tasks, significantly outperforming baseline models.
- Ensembling for Robustness: Bagging multiple BERT-Large models proved to be a powerful technique for reducing error caused by high variance, leading to substantial performance gains, especially in more challenging tasks like controversy detection.
- Subjectivity is Key: The authors acknowledge that humor is inherently subjective, and their approach to controversy detection (variance in annotator ratings) directly addresses this.
- Meta-Learning for Imbalance: For the controversy detection task, which suffered from class imbalance, meta-learning by tuning softmax thresholds was crucial for improving performance.
- Task-Specific Adaptations: Different BERT architectures and post-processing steps (like textual entailment framing or CNN layers) were explored, showing that task-specific adaptations can yield better results.
Key Results
The team "EndTimes" achieved competitive rankings:
- 4th in Humor Controversy Detection (F1 score of 0.6261).
- 8th in Humor Detection (F1 score of 0.9655).
- 19th in Average Offense Score prediction (RMS Error of 0.469).
- 40th in Average Humor Score prediction (RMS Error of 0.654).
The BERT-ENS model, their bagged ensemble, consistently outperformed individual BERT models across most tasks. For instance, in Humor Detection, BERT-ENS achieved an F1 score of 0.966, compared to BERT-L's 0.926.
Limitations and Future Work
- Limitations: The authors note that controversy detection is a challenging task due to its abstract nature and class imbalance. They also mention that changing the maximum sentence length didn't significantly impact results, suggesting potential for further optimization. The BERT-ENT approach, while explored, showed relatively poor performance, indicating its limitations for this specific task.
- Future Work: The authors suggest exploring other ensemble methods like stacking and blending. They also propose combining models with different hyperparameter values in more sophisticated ways to achieve even better results.
Summary updated