TL;DR
This paper introduces a deep multi-task learning (MTL) model for detecting and rating humor and offense in text. By leveraging a pre-trained transformer encoder with task-specific attention layers and an uncertainty-based loss weighting mechanism, their model achieves state-of-the-art results on the SemEval-2021 Task 7, outperforming single-task approaches.
What Problem Does This Paper Address?
Humor detection and understanding the nuances of offensive language are complex challenges in Natural Language Processing (NLP). While many existing approaches treat these as binary classification problems, the authors argue that humor exists on a spectrum and can be controversial. Similarly, offensive language can vary in intensity. This paper addresses the need for a unified system that can not only detect humor and offense but also rate their degrees and identify controversial humor, tackling these related tasks simultaneously.
Methods
The core of the proposed system is a deep Multi-Task Learning (MTL) model. It starts with a pre-trained transformer encoder (specifically, BERT and RoBERTa variants) to capture contextual word embeddings. Crucially, task-specific attention layers are applied on top of these embeddings to extract features relevant to each individual task. These task-specific representations, along with the pooled output from the encoder, are then fed into separate classification (for humor detection and controversy) and regression (for humor and offense rating) modules.
A key innovation is the use of MTL uncertainty loss weighting. Instead of simply averaging the losses from each task, this method adaptively weights them based on the estimated noise in each task's output. This allows the model to learn more effectively by prioritizing tasks with less inherent noise. The model is trained to minimize a combined loss function that includes binary cross-entropy for classification tasks and Root Mean Squared Error (RMSE) for regression tasks.
Datasets
The system was trained and evaluated on the SemEval-2021 Task 7 dataset. This dataset includes approximately 8000 sentences for training and 1000 for validation and testing. Sentences are labeled for humor detection (binary), humor degree (0-5), controversy (binary, based on annotator rating variance), and offense degree (0-5). The dataset was annotated by 20 individuals across a wide age range (18-70).
Key Insights and Ideas
- MTL for Nuance: Treating humor and offense as multi-faceted problems (detection, rating, controversy) and solving them jointly via MTL proves more effective than tackling them in isolation.
- Attention is Key: Task-specific attention layers significantly improve performance by allowing the model to focus on the most relevant parts of the text for each specific task.
- Uncertainty Matters: Adaptive weighting of task losses using uncertainty is crucial for effective MTL, allowing the model to better balance the contributions of related but distinct tasks.
- Transformer Power: Pre-trained transformer models, especially larger variants like RoBERTa-large, provide a strong foundation for capturing complex linguistic patterns necessary for these tasks.
Key Results
The proposed MTL model consistently outperformed its single-task counterparts across all sub-tasks. The best performance was achieved using the RoBERTa-large encoder. Specifically, the MTL model with RoBERTa-large achieved an F1-score of 0.9606 for humor detection (Task 1a), an RMSE of 0.4696 for humor rating (Task 1b), an F1-score of 0.6242 for controversy detection (Task 1c), and an RMSE of 0.5401 for offense rating (Task 2a). The paper also highlights that the inclusion of task-specific attention layers and uncertainty loss weighting contributed to performance gains.
Limitations and Future Work
The authors acknowledge that while their MTL model leverages signals from related tasks, there's potential to further improve by explicitly modeling the relationships between these tasks. Future work could involve exploring more sophisticated ways to capture these inter-task dependencies. Additionally, they suggest extending the model's capabilities beyond detection and rating to perform other related NLP tasks.
Summary updated