TL;DR
This paper presents SarcasmDet, a top-performing system for SemEval-2021 Task 7, which aims to detect and rate humor and offensiveness in text, considering demographic factors. The system leverages RoBERTa and BERT pre-trained models with a hard-voting ensemble technique, achieving strong results across multiple sub-tasks.
What Problem Does This Paper Address?
The increasing prevalence of online communication has brought to the forefront the challenges of automatically detecting nuanced linguistic phenomena like sarcasm, humor, and offensiveness. These forms of expression are often subtle, relying on context, cultural understanding, and even demographic factors to be fully interpreted. SemEval-2021 Task 7, "HaHackathon," specifically targets this by asking participants to not only detect humor and offensiveness but also to rate their intensity and controversiality, while also considering demographic influences. This task is crucial for developing more sophisticated and context-aware NLP systems that can better understand and moderate online content.
Methods
The SarcasmDet system is built upon powerful pre-trained language models, primarily RoBERTa and BERT. The authors experimented with various configurations of these models, including different sizes (base and large) and hyperparameters. A key technique employed is hard-voting ensemble, where the predictions from multiple well-performing models are combined to produce a final, more robust prediction. This ensemble approach proved instrumental in improving the system's overall performance. For Task 1c (Humor Controversy), a novel approach was used by leveraging the results from Task 1a (Humor Detection) and Task 1b (Average Humor Score) to predict controversy.
Datasets
The primary dataset used for this task was provided by the SemEval-2021 organizers for Task 7. This dataset contains approximately 10,000 rows of text data, divided into training, development, and evaluation sets. The data was collected by surveying US English native speakers across various ages, genders, political situations, and income levels. Each text was labeled based on four questions related to the specific tasks: humor detection, humor rating, humor controversy, and general offensiveness.
Key Insights and Ideas
- Ensemble Power: The hard-voting ensemble technique significantly boosted performance across all tasks, demonstrating the benefit of combining diverse model predictions.
- RoBERTa's Strength: RoBERTa, particularly the large-cased version, consistently showed strong performance, often outperforming other models on its own.
- Task Interdependence: For Task 1c, a clever approach of using outputs from Task 1a and 1b to predict controversy proved effective, highlighting the potential for multi-task learning or sequential prediction in related NLP tasks.
- Dataset Imbalance Impact: The paper notes that dataset imbalance, particularly in Task 1c, negatively impacted prediction accuracy, underscoring the importance of balanced datasets for robust model training.
Key Results
The SarcasmDet system achieved impressive rankings in the SemEval-2021 Task 7:
- Task 1a (Humor Detection): Ranked 4th with an F1-score of 0.9675.
- Task 1b (Average Humor Score): Ranked 10th with an RMSE of 0.5446.
- Task 1c (Humor Controversy): Ranked 3rd with an F1-score of 0.6270.
- Task 2 (Average Offensiveness Score): Ranked 10th with an RMSE of 0.4469.
The paper also details specific F1 and RMSE scores for individual models and the ensemble, showcasing the incremental improvements gained through their methodology.
Limitations and Future Work
The authors acknowledge that the imbalanced nature of the dataset for Task 1c led to a decrease in prediction performance. This suggests that future work could focus on more advanced techniques for handling imbalanced data, such as data augmentation or specialized loss functions. Additionally, while the paper mentions experimenting with various pre-trained models, further exploration into other state-of-the-art architectures or fine-tuning strategies could potentially yield even better results. The inclusion of demographic factors in the task also opens avenues for research into more personalized or context-aware humor and offense detection models.
Summary updated