UMUTeam at SemEval-2023 Task 11: Ensemble Learning applied to Binary Supervised Classifiers with disagreements

José Antonio García-Díaz, Ronghao Pan, Gema Alcaráz-Mármol, María José Marín-Pérez, Rafael Valencia-GarcíaACL 2023

TL;DR

This paper presents UMUTeam's approach to the SemEval-2023 Task 11 (Le-Wi-Di), which focuses on building classifiers that account for annotator disagreements. They explored ensemble learning with large language models (LLMs) and linguistic features, achieving a respectable 18th place out of 30 participants, despite not directly incorporating disagreement information into their models.

What Problem Does This Paper Address?

The core challenge addressed is the development of supervised machine learning models that can effectively handle subjective tasks where human annotators often disagree. Traditional supervised learning assumes a single ground truth for each data point, but in many real-world scenarios, especially those involving opinions, sentiment, or nuanced interpretations, annotators may have validly different perspectives. The Le-Wi-Di task specifically aims to create a unified framework for learning from these disagreements, acknowledging that a model's performance should ideally reflect the inherent ambiguity in the data. This is particularly relevant for tasks like hate speech detection, misogyny detection, and other forms of offensive language identification, where subjectivity plays a significant role.

Methods

UMUTeam's approach is grounded in ensemble learning, combining the outputs of multiple models to achieve a more robust prediction. Their pipeline involves:

  1. Data Cleaning: Basic preprocessing to remove noise like hyperlinks, mentions, and hashtags.
  2. Dataset Splitting Strategy: Evaluating two strategies: training a single model on a merged dataset (multi-task learning) versus training separate classifiers for each individual dataset.
  3. Feature Extraction:
    • Large Language Models (LLMs): Fine-tuning three LLMs (mBERT, multilingual BERT, and XLM) on the task data. Sentence embeddings are extracted from the [CLS] token.
  4. Linguistic Features (LFs): Utilizing features from UMUTextStats, a tool designed for Spanish text analysis that captures various linguistic categories. While initially explored, LFs were eventually found to downplay overall results when combined with LLM embeddings.
  5. Model Training: Training supervised classifiers on the extracted features. For LLMs, this involved fine-tuning with hyperparameter optimization.
  6. Ensemble Learning: Combining the predictions from individual models using several strategies:
  7. Knowledge Integration (KI): Training a new classifier on the sentence embeddings from each LLM.
  8. Ensemble Strategies: * Highest Probability (HIGHEST): Taking the maximum probability across models.
  9. Average of Probabilities (MEAN): Averaging the predicted probabilities.
  10. Hard Voting (MODE): Selecting the most frequent predicted label.

A key aspect of their methodology is the absence of direct incorporation of disagreement information during the training phase, which they acknowledge as a limitation.

Datasets

The task involved four normalized, multilingual datasets focused on subjective tasks:

  • HS-Brexit: Annotations for hate speech and abusive language related to Brexit.
  • ArMIS: Tweets in Arabic annotated for misogyny detection.
  • ConvAbuse: English dialogues between users and conversational agents, annotated for offensiveness.
  • MultiDomain Agreement (MD-Agreement): English tweets concerning BLM, elections, and Covid-19, annotated for offensiveness.

The paper notes significant differences in dataset size and class balance, with Brexit having a notably imbalanced distribution of positive labels.

Key Insights and Ideas

  • Ensemble Learning as a Strong Baseline: Even without directly modeling disagreement, ensemble methods (especially averaging probabilities) provided strong results, outperforming individual models and many other participants. This suggests that combining diverse model outputs is a robust strategy for subjective tasks.
  • LLMs are Powerful, but Not a Silver Bullet: The fine-tuned LLMs (mBERT, multilingual BERT, XLM) formed the backbone of their strong performance, demonstrating their capability in capturing nuanced linguistic information.
  • Linguistic Features' Limited Impact: The authors found that linguistic features from UMUTextStats, while useful in other contexts, did not significantly improve performance when combined with LLM embeddings for this specific task, and in some cases, even degraded results.
  • The Importance of Disagreement (Implied): The paper's ranking (18th out of 30) and the discussion about the differences between soft (cross-entropy) and hard (F1-score) metrics implicitly highlight the value of disagreement information. Their ability to achieve decent results without it suggests that incorporating it could lead to even better performance.

Key Results

  • Official Ranking: UMUTeam ranked 18th out of 30 participants on the official leaderboard, with an average cross-entropy of 18.75.
  • F1-Score Performance: When considering the F1-score metric (which they found to be a better indicator), they achieved 13th place.
  • Best Performing Strategy: The Knowledge Integration (KI) strategy achieved the best F1-score (82.066%) on the merged dataset with their custom validation split. For the official submission, they used an ensemble learning strategy based on averaging probabilities, achieving an F1-score of 0.8303.
  • Dataset-Specific Performance:
    • The best individual model performance was achieved by XLM on the ArMIS dataset (74.00% F1).
  • LF performed best on HS-Brexit (78.22% F1), though this was attributed to the dataset's extreme imbalance.
  • KI was best for MD-Agreement (80.01% F1).
  • EL (MODE) was best for ConvAbuse (91.189% F1).

Limitations and Future Work

  • Lack of Disagreement Incorporation: The most significant limitation acknowledged is that their models did not directly incorporate information about annotator disagreements during training. They believe this is a crucial aspect for improving performance on subjective tasks.
  • Mistake in Submission: The team made a mistake in generating their initial runs, leading them to submit only 3 runs to the official leaderboard instead of the allowed 5.
  • Future Work: The authors explicitly state their intention to incorporate disagreement information into their pipeline in future work, recognizing its potential to significantly improve their models' performance. They also suggest that their approach, which achieved good results without explicit disagreement modeling, could serve as a more realistic baseline for future research.

Summary updated

BibTeX
@inproceedings{garciadiaz2023umuteam,
  author = {José Antonio García-Díaz and Ronghao Pan and Gema Alcaráz-Mármol and María José Marín-Pérez and Rafael Valencia-García},
  title = {{UMUTeam at SemEval-2023 Task 11: Ensemble Learning applied to Binary Supervised Classifiers with disagreements}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure