eevvgg at SemEval-2023 Task 11: Offensive Language Classification with Rater-based Information

Ewelina GajewskaACL 2023

TL;DR

This paper tackles the challenge of offensive language detection by moving beyond simple majority voting. The authors propose a BERT-based model that leverages individual annotator disagreements and their characteristics to improve classification accuracy across multiple datasets. Their approach achieved a respectable 11th place overall in SemEval-2023 Task 11, highlighting the value of incorporating nuanced annotator information.

What Problem Does This Paper Address?

Traditional text classification often relies on a "majority rules" approach, assuming a single correct label for each data point. However, subjective tasks like identifying offensive language are inherently prone to disagreement among human annotators. These disagreements aren't just noise; they can contain valuable information about the nuances of language and individual interpretation. This paper addresses the gap in effectively utilizing this disagreement information to build more robust offensive language detection models. It builds on prior work that explored the impact of annotator demographics and the use of disaggregated data, aiming to create a more sophisticated understanding of offensive language.

Methods

The core of the proposed system is a BERT-based model, specifically BERTweet for English and BERT-base for Arabic. The authors fine-tune these pre-trained models on each dataset. What's novel is their approach to incorporating "rater-based information." This includes:

  • Partially Disaggregated Data: Instead of solely using majority-voted labels, instances where annotators disagreed were used with both the "offensive" and "non-offensive" labels in the training set. This allows the model to learn from conflicting signals.
  • Additional Features: Beyond the standard text embeddings from BERT, the model incorporates engineered features derived from annotator behavior and characteristics. These features vary by dataset but can include:
  • Keywords identified as specific to offensive categories (e.g., hate speech, misogyny).
  • Rater agreement metrics (e.g., Cohen's kappa) to model individual rater reliability against the majority.
  • Annotations for aggressive and offensive language, and rater group identity (for HS-Brexit).

The model architecture involves feeding BERT encodings into a first dense layer, followed by dropout, then concatenating additional engineered features before a second dense layer and a final classification layer.

Datasets

The paper evaluates the proposed system on four datasets from SemEval-2023 Task 11:

  • HS-Brexit: Annotated with hate speech.
  • ConvAbuse: Focuses on abusiveness detection in conversations.
  • ArMIS: Deals with misogyny and sexism detection in Arabic texts.
  • MD-Agreement: A larger dataset (over 800 annotators) focused on general offensiveness detection.

The MD-Agreement dataset is notable for its significantly larger pool of annotators compared to the other datasets, making it a more challenging and interesting case for studying disagreement.

Key Insights and Ideas

  • Disagreement is Informative: The paper strongly suggests that treating annotator disagreements as a valuable signal, rather than just noise, can significantly improve offensive language detection.
  • Rater Behavior Matters: Incorporating features derived from how annotators behave (e.g., their agreement patterns, keyword usage) provides crucial context that text alone might miss.
  • Dataset Scale Impacts Strategy: The authors found that the MD-Agreement dataset, with its vast number of annotators, presented unique challenges and opportunities, leading to a more complex feature engineering approach for that specific task.
  • BERT's Adaptability: The use of pre-trained BERT models, fine-tuned with these novel features, demonstrates the power and flexibility of these architectures for nuanced NLP tasks.

Key Results

The eevvgg team achieved an overall ranking of 11th place across the four datasets in SemEval-2023 Task 11. They performed particularly well on the MD-Agreement dataset, securing 6th place.

Quantitatively, the paper reports:

  • Micro-averaged F1-scores that are significantly higher than the baseline for most datasets, with notable improvements on ConvAbuse (+76%) and MD-Agreement (+55%).
  • Macro-averaged F1-scores of .77, .68, .73, .87, and .79 for HS-Brexit, ArMIS, ConvAbuse, and MD-Agreement, respectively.
  • The system achieved good performance for both classes, especially on ConvAbuse and MD-Agreement, with macro-F1 scores approaching 0.9 and 0.8.
  • Confusion matrices reveal that while the system performs well, it tends to underperform in correctly classifying positive offensive cases (predicting "0" when the true label is "1").

Qualitatively, the paper highlights that while sensitive or slur words are often correctly identified, challenging cases involve subtle offensiveness or negative framing that can still trick the model.

Limitations and Future Work

The authors acknowledge that their system, while effective, is relatively simple in its architecture and employed features. They suggest several avenues for future work:

  • Pre-training on Larger Datasets: Exploring pre-training on even larger, domain-specific datasets (like hate speech corpora) could further enhance performance. They mention that an initial experiment with HateBERT did not show improvement, suggesting careful selection and integration are key.
  • Individual Rater Prediction: Extending the approach to predict labels for individual raters could pave the way for personalized hate speech detection and filtering systems.
  • Challenging Cases: Further research is needed to address the tricky cases where subtle negative framing or disguised offensive language is used, and where the system incorrectly classifies instances.

Summary updated

BibTeX
@inproceedings{gajewska2023eevvgg,
  author = {Ewelina Gajewska},
  title = {{eevvgg at SemEval-2023 Task 11: Offensive Language Classification with Rater-based Information}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure