Tackling Racial Bias in Automated Online Hate Detection: Towards Fair and Accurate Classification of Hateful Online Users Using Geometric Deep Learning

Zo Ahmed, Bertie Vidgen, Scott A. HalearXiv 2021 · arXiv

TL;DR

This paper demonstrates that incorporating social network information via Geometric Deep Learning (specifically GraphSAGE) significantly improves the accuracy and fairness of automated "hateful user" detection. By leveraging the context of a user's social neighborhood, the researchers were able to eliminate the common algorithmic bias that misclassifies African-American Vernacular English (AAVE) as hateful, achieving a 0% false positive rate for African-American users in their best-performing model.

What Problem Does This Paper Address?

Automated hate speech detection is a minefield of bias. Most existing systems focus on analyzing individual pieces of content (e.g., a single tweet). However, these models often struggle with polysemy—words that have different meanings depending on the speaker's identity and context. A notorious example is the term "nigga," which is frequently used innocuously or self-referentially within the African-American community but is often flagged as "toxic" by content-blind classifiers. This leads to a high False Positive Rate (FPR) for African-American users, effectively suppressing their speech.

Furthermore, social media platforms are shifting from simply moderating content to identifying "hateful users." This paper addresses the dual challenge of making these user-level classification systems both more accurate and fairer, specifically targeting the reduction of racial bias.

Methods

The researchers compared traditional machine learning baselines against Geometric Deep Learning (GDL).

  1. Race Inference: To evaluate fairness, the authors used a distantly supervised probabilistic topic model to identify users likely to be African-American based on their use of AAVE. This was followed by manual verification to ensure a high-quality subset for testing.
  2. Classification Models: They tested seven models:
  3. Baselines: Logistic Regression (LR), Support Vector Machines (SVM), and Artificial Neural Networks (ANN) using only textual and user-level features.
  4. Feature-Engineered ANN: An ANN that included manually engineered network features (like centrality measures).
  5. Geometric Deep Learning: Three variants of GraphSAGE (mean aggregation, max pooling, and self-attention). Unlike "shallow" embedding methods like node2vec, GraphSAGE is inductive—it learns functions to aggregate information from a node's neighborhood, allowing it to generalize to new, unseen users without retraining the entire graph.

The primary fairness metric used was predictive equality, which requires that the False Positive Rate be balanced across different demographic groups.

Datasets

The study utilized a Twitter retweet network dataset originally prepared by Ribeiro et al. (2018):

  • Total Network: 100,386 users (nodes) and over 2.2 million edges.
  • Annotated Subset: 4,972 users manually labeled as either "hateful" or "normal."
  • Fairness Subset: 136 users identified and verified as African-American (AA) to test for disparate impact.

Key Insights and Ideas

  • Context is King: The "aha moment" here is that a user's social network provides the missing context that text alone cannot. Because hateful users tend to be homophilous (they cluster together in dense, resilient networks), the model can learn to distinguish between a user using "edgy" language in a normal community versus a user embedded in a hateful cluster.
  • Beyond Guilt-by-Association: While there is a risk of "guilt-by-association," the authors found that GDL actually reduced bias. By dynamically learning network representations, the model realized that African-American users, despite using certain keywords, did not share the network characteristics of hateful clusters.
  • No Fairness-Accuracy Trade-off: Contrary to the common belief that fairness constraints hurt performance, the best-performing model (GraphSAGE with mean aggregation) was both the most accurate and the fairest.

Key Results

The quantitative results highlight a clear victory for Geometric Deep Learning:

  • Overall Accuracy: GraphSAGE (mean aggregation) achieved the highest AUC of 90.8% and the best F1-Score of 57.5%.
  • Fairness Breakthrough: As shown in the fairness evaluation (Figure 3), the baseline ANN without network features had a 20.3% False Positive Rate for African-American users. GraphSAGE (mean aggregation) brought this False Positive Rate down to 0.0%.
Table 1: Accuracy Metrics (Maximum columnar values in bold)
Table 2: Fairness Evaluation Metrics (African-American denoted as AA, False positive denoted as FP)

The confusion matrix for the best model (Figure 1) shows that while the model is excellent at detecting hateful users, it still tends to be conservative, with more false positives (492) than false negatives (126) across the entire dataset. However, the crucial finding is that these false positives are no longer concentrated in the African-American subset.

Figure 1: Confusion matrix of predictions from best per- forming model: GraphSAGE with mean aggregation

Limitations and Future Work

  • Precision vs. Recall: While the AUC is high, the precision remains relatively low (~46%). The models are good at finding hateful users but still flag many normal users as hateful.
  • Data Scarcity: The number of annotated hateful users (544) and African-American users (136) is small. The authors suggest that more annotated data would likely improve precision.
  • Ground Truth Reliability: Qualitative error analysis revealed that a significant portion of "errors" were actually due to human annotation mistakes in the original dataset.
  • Future Directions: The authors suggest exploring heterogeneous graphs (treating users and tweets as different types of nodes) and incorporating more advanced language models like BERT, which were not feasible for this specific dataset due to missing raw text for some users.

Summary updated

Abstract

Online hate is a growing concern on many social media platforms and other sites. To combat it, technology companies are increasingly identifying and sanctioning `hateful users' rather than simply moderating hateful content. Yet, most research in online hate detection to date has focused on hateful content. This paper examines how fairer and more accurate hateful user detection systems can be developed by incorporating social network information through geometric deep learning. Geometric deep learning dynamically learns information-rich network representations and can generalise to unseen nodes. This is essential for moving beyond manually engineered network features, which lack scalability and produce information-sparse network representations. This paper compares the accuracy of geometric deep learning with other techniques which either exclude network information or incorporate it through manual feature engineering (e.g., node2vec). It also evaluates the fairness of these techniques using the `predictive equality' criteria, comparing the false positive rates on a subset of 136 African-American users with 4836 other users. Geometric deep learning produces the most accurate and fairest classifier, with an AUC score of 90.8\% on the entire dataset and a false positive rate of zero among the African-American subset for the best performing model. This highlights the benefits of more effectively incorporating social network features in automated hateful user detection. Such an approach is also easily operationalized for real-world content moderation as it has an efficient and scalable design.
BibTeX
@misc{ahmed2021tackling,
  author = {Zo Ahmed and Bertie Vidgen and Scott A. Hale},
  title = {{Tackling Racial Bias in Automated Online Hate Detection: Towards Fair and Accurate Classification of Hateful Online Users Using Geometric Deep Learning}},
  year = {2021},
  eprint = {2103.11806},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2103.11806},
}
Figure