GHOST at SemEval-2021 Task 5: Is explanation all you need?

Kamil Pluciński, Hanna KlimczakACL 2021

TL;DR

This paper explores whether explainable AI (XAI) methods can achieve comparable performance to supervised models for toxic span detection, a task crucial for online content moderation. While supervised BERT-based models ultimately performed best, XAI approaches like SHAP showed promise in offering transparency and potentially reducing data labeling costs.

What Problem Does This Paper Address?

The increasing prevalence of social media has brought with it a surge in online toxicity, including cyberbullying and hate speech. While automatic detection of toxic comments is improving, many advanced models lack transparency, making it difficult for human moderators to trust their predictions. This paper tackles the challenge of "Toxic Spans Detection," which goes beyond simply classifying a whole comment as toxic. Instead, it aims to pinpoint the exact words or phrases that contribute most to the toxicity. This granular approach is vital for effective moderation and for understanding why a comment is flagged. The core question is whether explainable AI (XAI) techniques, designed to make model decisions understandable, can perform as well as traditional supervised learning methods for this specific, fine-grained task.

Methods

The authors investigated three main approaches:

  • Orthogonal LSTM with Attention: This method uses a Long Short-Term Memory (LSTM) network enhanced with an attention mechanism. The novelty lies in an "orthogonalization" technique applied to the LSTM's hidden states, aiming to make the attention vectors more distinct and thus better indicators of important words. This is a model-specific XAI approach.
  • SHAP (SHapley Additive exPlanations) with BERT: SHAP is a model-agnostic framework that uses Shapley values to explain the predictions of any machine learning model. Here, it's applied to a pre-trained BERT model fine-tuned for toxic comment classification. The goal is to understand which words BERT considers most influential in its toxicity predictions.
  • Supervised Token Classification with BERT and Extensions: This is the traditional supervised learning approach. A BERT model is fine-tuned to directly classify each token (word or sub-word) as toxic or not. The paper also explores variations like using different BERT-based models (XLNet, RoBERTa, ELECTRA, SqueezeBERT) and techniques to handle long input sequences and data augmentation to combat overfitting.

Datasets

The primary dataset used for training and evaluation was the CivilComments dataset. For the Toxic Spans Detection task specifically, the authors utilized the dataset from SemEval-2021 Task 5.

Key Insights and Ideas

  • XAI vs. Supervised Learning Trade-off: While XAI methods aim for transparency, they don't necessarily outperform specialized supervised models designed for the exact task. The supervised BERT approach, trained specifically for token-level toxicity, achieved the best results.
  • Attention as an Explanation: The paper highlights the ongoing debate about whether attention mechanisms truly provide reliable explanations. While attention can point to potentially toxic words, it might lead to more false negatives compared to methods like SHAP.
  • SHAP's Promise: The SHAP framework, being model-agnostic, demonstrated a surprisingly strong performance, outperforming the attention-based LSTM. This suggests that model-agnostic explanations can be a valuable tool for understanding complex models.
  • Data Labeling Costs: A significant advantage of XAI methods is their potential to reduce the need for extensive, fine-grained data labeling, which is a major bottleneck for supervised token classification.
  • Handling Long Inputs: BERT's inherent limitation of a 512-token input window required creative solutions like reformatting inputs by adding whole sentences, which impacted performance slightly.
  • Subtle Toxicity: The inclusion of spaces as potentially toxic elements (due to how they were labeled in the dataset) showed a small but positive impact on performance, indicating the nuances of real-world data labeling.

Key Results

  • The ensemble of BERT-based models achieved the highest span-level F1 score, placing 13th out of 91 solutions in the SemEval-2021 task.
  • The supervised BERT model fine-tuned for token classification achieved a span-level F1 score of 0.672 on the validation set.
  • The SHAP approach applied to a BERT model achieved a span-level F1 score of 0.5987 on the validation set.
  • The Orthogonal LSTM with Attention achieved a span-level F1 score of 0.4970 on the validation set.
  • The paper notes that models with more parameters (like XLNet, RoBERTa) generally performed better than BERT, but were also more prone to overfitting.

Limitations and Future Work

  • XAI Performance Gap: The XAI approaches (LSTM with attention and SHAP) did not reach the performance levels of the supervised BERT model for this specific task.
  • Threshold Tuning: Both XAI methods required careful threshold tuning to determine which attention scores or Shapley values indicated toxicity, which could be seen as a form of manual tuning.
  • False Positives: The attention-based method showed a higher number of false negatives compared to SHAP, suggesting that while attention might highlight toxic words, it might not be sufficient for precise identification.
  • Future Work: The authors suggest further investigation into XAI methods for improving transparency and understanding in comment moderation. They also propose exploring how these methods can be adapted for tasks where binary decision models are sufficient, potentially reducing data preparation costs. The need for further research to assess the performance of these methods for tasks specified in a more detailed way is also mentioned.

Summary updated

BibTeX
@inproceedings{plucinski2021ghost,
  author = {Kamil Pluciński and Hanna Klimczak},
  title = {{GHOST at SemEval-2021 Task 5: Is explanation all you need?}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure