Lexicools at SemEval-2023 Task 10: Sexism Lexicon Construction via XAI

Pakawat Nakwijit, Mahmoud Samir, Matthew PurverACL 2023

TL;DR

This paper tackles the challenge of detecting online sexism by developing explainable lexicons using XAI techniques like Shapley values. Their approach not only constructs high-quality lexicons but also demonstrates that these lexicons, when used in simpler models, can outperform complex fine-tuned LLMs on certain sexism detection tasks, offering a more interpretable and potentially less biased solution.

What Problem Does This Paper Address?

Online sexism is a pervasive issue, and while automated detection methods are crucial due to the sheer volume of social media content, they often function as "black boxes." This lack of transparency makes it difficult to understand why a model flags something as sexist, hindering trust, bias identification, and model improvement. SemEval-2023 Task 10 (EDOS) specifically aims to develop explainable sexism detection systems. This paper addresses this by focusing on lexicon-based approaches, which are inherently more interpretable, and leveraging XAI techniques to build and refine these lexicons.

Methods

The authors employ a multi-stage approach for lexicon construction and sexism detection:

  • Lexicon Construction: They experiment with two methods:
  • Pointwise Mutual Information (PMI): A statistical measure to assess word association.
  • Shapley Values (via SHAP): A game-theoretic approach to attribute the contribution of each word to a model's prediction (in this case, a fine-tuned BERT model from Task A). This is a novel application for lexicon construction.
  • Lexicon Augmentation: To address limited vocabulary coverage, they augment their lexicons using:
  • An unannotated corpus (from Gab, filtered by their Task A model).
  • Large Language Models (LLMs) like BERTweet and GPT-J, using fill-in-the-blank and prompt-based generation techniques.
  • Classifier Integration: The constructed and augmented lexicons are integrated into three types of classifiers:
  • A simple lexicon-based classifier (counting word occurrences).
  • Bag-of-Words (BoW) logistic regression.
  • Fine-tuned LLMs (specifically BERT).

Datasets

The study utilizes datasets provided by the SemEval-2023 Task 10 (EDOS) organizers. This includes labeled data for training and evaluation, as well as an unannotated corpus. No new benchmark datasets were introduced.

Key Insights and Ideas

  • Shapley Values for Lexicon Quality: The Shapley value approach, when augmented, yields lexicons that are both quantitatively and qualitatively superior to those derived solely from PMI. This suggests XAI can directly improve the quality of linguistic resources.
  • Augmentation is Crucial: Expanding lexicon vocabulary through augmentation significantly improves performance, especially in low-resource settings, by reducing the "missing rate" (sentences with no lexicon overlap).
  • Simpler Models Can Outperform: Surprisingly, a basic lexicon-based classifier, simply counting word occurrences, can outperform a fine-tuned BERT model on certain tasks (Task B/C) and even a BoW logistic regression, highlighting the power of well-constructed lexicons.
  • XAI for Bias Detection: The authors use XAI to identify model limitations, such as poor handling of negation and biases against specific demographic groups (e.g., LGBTQ+ individuals), demonstrating XAI's utility beyond just feature importance.
  • Urban Slang Dominance: Urban slang is identified as a significant contributor to sexism predictions, emphasizing the need for models to adapt to evolving language.

Key Results

  • Lexicon-Based Classifier Performance: The best lexicon-based classifier (Shapley + BERTweet augmentation) achieved F1-scores of 53.34% and 27.31% on the official blind test sets for tasks B and C, respectively.
  • Outperforming Baselines: The lexicon-based classifier outperformed BoW logistic regression in all metrics for tasks B and C. It also outperformed fine-tuned BERT in Task C.
  • Augmentation Effectiveness: Augmentation reduced the missing rate in lexicon-based classifiers to 0% and improved F1 scores by up to 17.8% and 19.5% for tasks B and C, respectively. BERTweet augmentation was found to be the most effective.
  • Feature Selection: Using lexicons as features in logistic regression reduced the number of model parameters by 35.7% while maintaining high performance.
  • Robustness: The model demonstrated robustness against noise, maintaining predictions for 95.625% of sentences when random "funny" words were added.

Limitations and Future Work

  • Context Dependency: Lexicon-based approaches struggle with context-dependent meanings like idioms, sarcasm, and multi-word expressions.
  • PMI Limitations: PMI can overvalue relatedness in low-frequency events, potentially leading to faulty lexicons.
  • Shapley Limitations: Shapley values can lead to unrealistic conclusions when features are correlated, and their explainability is tied to the underlying model's performance and biases.
  • Dataset Artifacts: The lexicons might pick up on dataset artifacts rather than genuine indicators of sexism, leading to false positives.
  • Negation Handling: Fine-tuned LLMs still show limitations in correctly handling negation.
  • Bias in Training Data: The study highlights potential under-representation of vulnerable groups in training data, leading to biased model behavior.
  • Future Work: The authors suggest the need for more advanced techniques to improve LLM fine-tuning with lexicons and further investigate how to mitigate dataset artifacts and biases. They also emphasize the importance of continuous vocabulary updates to capture evolving language.

Summary updated

BibTeX
@inproceedings{nakwijit2023lexicools,
  author = {Pakawat Nakwijit and Mahmoud Samir and Matthew Purver},
  title = {{Lexicools at SemEval-2023 Task 10: Sexism Lexicon Construction via XAI}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure