Detecting and Reducing Bias in a High Stakes Domain

Ruiqi Zhong, Yanda Chen, Desmond Patton, Charlotte Selous, Kathy McKeown

Abstract

Machine learning models used to detect gang-related aggression on social media often rely on spurious correlations, such as neutral stop words, leading to biased predictions in high-stakes environments. This paper investigates these biases and proposes a novel training method that aligns a model's attention mechanism with human-provided rationales. Our results demonstrate that this approach significantly reduces reliance on non-semantic features and improves model robustness without compromising predictive accuracy.

Introduction

In the domain of "NLP for Social Good," researchers leverage machine learning to identify social media signals of impending physical violence. In urban centers like Chicago, community outreach workers utilize these tools to intervene in "internet banging"—online provocations that frequently escalate into real-world harm. However, the high-stakes nature of this task carries significant risks: misclassifying benign communication as "aggression" can further marginalize and criminalize vulnerable populations, particularly young Black and Latino men. This study identifies a critical failure in state-of-the-art models: a reliance on spurious features rather than semantic intent.

Identifying the "Stop Word" Trap

To understand the decision-making process of existing models, the authors performed a post-mortem analysis on a standard Convolutional Neural Network (CNN) using the Leave-One-Out (LOO) method. This technique identifies which specific words most heavily influence a model's classification. The analysis revealed a startling trend: baseline models frequently relied on common stop words—such as "a," "on," "into," and "of"—to predict aggression.

Unigram influence across models, highlighting the disproportionate reliance on stop words in baseline architectures.

As shown in the data above, these words, which carry no semantic weight regarding violence, were treated as high-signal features. This "stop word trap" suggests that models may simply be memorizing the idiosyncratic linguistic patterns of specific demographic groups rather than learning the concept of aggression itself. Adversarial tests confirmed this vulnerability, showing that inserting the word "on" into a non-aggressive tweet could flip the model's prediction to "aggression" up to 5% of the time.

Methodology: Human-Centric Attention

To mitigate these biases, the researchers transitioned from a CNN architecture to a Long Short-Term Memory (LSTM) network equipped with an attention mechanism. The core innovation, Trained Attention with Rationale, involves three key steps:

  1. Rationale Annotation: Domain experts (social workers) annotated a subset of the dataset, highlighting the specific words that justified an "aggression" label.
  2. Ground Truth Attention: A target attention distribution ($A^*$) was established, concentrating weight solely on these expert-highlighted rationales.
  3. Loss Function Modification: The training process was modified to include a KL Divergence loss term. This penalizes the model when its internal attention distribution deviates from the human-provided "ground truth," effectively teaching the model to "look" at the same features as a human expert.

Results and Performance

The "LSTM + Rationale" model demonstrated superior robustness and alignment with human reasoning. In adversarial tests, the rationale-trained model was significantly less likely to flip its prediction when neutral stop words were inserted compared to the baseline.

Performance comparison of various models across F1 metrics, showing that rationale-based training maintains state-of-the-art accuracy.

Crucially, as illustrated in the performance table, these gains in robustness did not come at the cost of accuracy. The LSTM + Rationale model achieved an Ensemble Macro F1 of 69.8, slightly outperforming the previous state-of-the-art (69.4). Furthermore, the model's most influential words matched human rationales much more closely, achieving a Rationale Rank of 0.86 compared to 1.42 for the baseline.

Conclusion

This research highlights the danger of treating high-accuracy models as black boxes in sensitive domains. By incorporating human expertise directly into the model's attention mechanism, we can create AI systems that are not only more accurate but also more justifiable and less prone to harmful biases. Future work will focus on capturing more complex contextual cues and neighborhood-specific pragmatics that go beyond simple word-level rationales.

Summary updated

BibTeX
@inproceedings{zhong2019detecting,
  author = {Ruiqi Zhong and Yanda Chen and Desmond Patton and Charlotte Selous and Kathy McKeown},
  title = {{Detecting and Reducing Bias in a High Stakes Domain}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=YPvu9ipomY},
}
Figure