TL;DR
This paper investigates biases in hate speech detection models, revealing that they can learn to associate marginalized groups with negative content for the wrong reasons. The authors introduce a new bias metric, "systematic offensive stereotyping" (SOS), and demonstrate its effectiveness in capturing biases in word embeddings that impact hate speech detection performance and fairness.
What Problem Does This Paper Address?
Automated hate speech detection models are crucial for creating safer online environments. However, recent research highlights that these models can exhibit biases, leading them to make correct predictions for incorrect reasons. This paper addresses the critical gap in understanding how these biases manifest and what their causal effect is on the performance and fairness of hate speech detection models. Specifically, it moves beyond simply identifying social biases (like gender or racial bias) to explore "systematic offensive stereotyping" (SOS) – the association of offensive terms with marginalized groups – and its intersectional nature. The authors also tackle the limitation that previous studies often show correlation but not causation between biases in word embeddings and downstream tasks.
Methods
The paper proposes a multi-faceted approach to investigate bias in hate speech detection:
- Model Performance Analysis: The authors first establish a baseline by comparing the performance of BERT against traditional RNN models (LSTM, Bi-LSTM) on various hate speech and abuse detection datasets.
- Bias Measurement (SOS): A novel metric, Systematic Offensive Stereotyping (SOS), is introduced. It quantifies the association between swear words and words describing marginalized groups (e.g., ethnicity, gender, sexual orientation) within word embeddings. This is achieved by calculating the cosine similarity between an average vector of swear words and the vectors of identity terms, normalized to a [0, 1] range.
- Intersectional Bias Measurement: The paper outlines plans to create an intersectional bias dataset (focusing on gender and race) and adapt existing methods to measure intersectional biases in contextual word embeddings.
- Causal Inference: To move beyond correlation, the authors propose using counterfactual causal inference to measure the causal influence of social and intersectional biases in word embeddings on hate speech detection tasks. This involves comparing model predictions with and without debiased embeddings.
- Fairness Measurement: The paper plans to measure unfairness in hate speech detection models by comparing false positive rates (FPR) between marginalized and non-marginalized groups.
Datasets
The research utilizes several publicly available datasets for training and evaluation:
- Hate Speech/Abuse Detection Datasets: Kaggle-insults, Twitter-sex, Twitter-rac, HateEval, WTP-agg, WTP-tox.
- POS Tagging Dataset: An English POS tags classification dataset for fine-tuning BERT.
- Bias Datasets (for future work): Plans to create an intersectional bias dataset similar to SteroSet and CrowS-Pairs.
- Targeted Hate Speech Datasets (for future work): ETHOS, MLMa, Jigsaw, MIT, SBIC.
Key Insights and Ideas
- Beyond Social Bias: The research highlights that biases in hate speech detection models extend beyond traditional social biases to include "systematic offensive stereotyping" (SOS), where offensive language is directly linked to marginalized groups.
- Word Embeddings Encode Harmful Associations: The SOS metric reveals that certain word embeddings, particularly those trained on social media data (like Glove-Twitter, UD, Chan), exhibit stronger associations between offensive terms and marginalized groups compared to those trained on general text (like Word2Vec, Glove-WK).
- Causality Matters: The paper emphasizes the need to move from correlational studies to causal inference to truly understand the impact of biases on model performance and fairness.
- BERT's Reliance on Syntactic Biases: Analysis of BERT's attention weights and feature importance suggests that the model may rely on syntactic biases and "shortcuts" within datasets rather than deep semantic understanding for hate speech detection.
- SOS as a Reflective Bias Metric: The proposed SOS metric shows a stronger correlation with real-world online abuse statistics compared to established bias metrics like WEAT, RNSB, and RND, suggesting it better captures relevant biases.
Key Results
- BERT Performance: BERT outperforms traditional RNN models (LSTM, Bi-LSTM) on various hate speech and abuse detection datasets, achieving the highest F1 scores.
- Attention Weights Not Meaningful for Prediction: Despite differences in attention patterns between fine-tuned and non-fine-tuned BERT, attention weights do not show a linear correlation with token importance, suggesting they don't directly explain model predictions.
- Syntactic Bias in BERT: Analysis of POS tag importance scores indicates that BERT captures syntactic biases in datasets, with pronouns being the most informative tag for hate speech detection, suggesting reliance on superficial features.
- SOS Bias Correlation: The SOS bias metric demonstrates a positive correlation with the F1 scores of deep learning models on hate speech detection tasks, indicating that SOS bias in word embeddings influences model performance.
- Word Embedding Differences: Word embeddings trained on social media data show a stronger SOS bias towards marginalized groups compared to those trained on general corpora.
Limitations and Future Work
- Western-Centric Perspective: The study acknowledges a limitation in focusing primarily on a Western societal perspective for bias measurement.
- Limited Scope of Intersectional Bias: The current work on intersectional bias focuses specifically on gender and race, with plans to expand.
- English Language Focus: The research is limited to English language models and datasets.
- Supervised Models Only: The study focuses on the influence of bias on supervised machine learning models for hate speech detection.
- Future Work: The authors plan to:
- Create an intersectional bias dataset.
- Measure intersectional bias in contextual word embeddings.
- Use counterfactual causal inference to measure the causal influence of bias on hate speech detection performance and fairness.
- Develop more effective and targeted debiasing techniques.
Summary updated