Detecting and Reducing Bias in a High Stakes Domain

Ruiqi Zhong, Yanda Chen, Desmond Patton, Charlotte Selous, Kathy McKeown

Abstract: Ruiqi Zhong, Yanda Chen, Desmond Patton, Charlotte Selous, Kathy McKeown. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

2019.

Introduction

In the intersection of machine learning and social justice, the detection of aggression in social media posts from gang-involved youth presents a high-stakes challenge. While automated tools are designed to assist community outreach workers in preventing street violence, current state-of-the-art models often rely on biased linguistic shortcuts. This research reveals that models can be "right for the wrong reasons," potentially leading to the over-criminalization of vulnerable populations by misinterpreting neutral language as aggressive.

The Problem of Spurious Correlations

Accuracy, typically measured by F1-score, can be a deceptive metric in high-stakes domains like criminal justice. If a model predicts "aggression" based on specific slang or common function words (stop words) rather than intent, it codifies bias rather than detecting violence. The authors investigated tweets from Chicago youth and found that previous models picked up on "spurious correlations"—patterns with no semantic link to aggression. Such models might unfairly flag innocent users, leading to harmful interventions for Black and Latino youth.

Methodology

The researchers developed a multi-step approach to identify and mitigate these biases:

  1. Locating Bias: Using a "leave-one-out" (LOO) method, they found that in over 10% of cases, models relied heavily on stop words like "a," "on," "into," and "of" to classify tweets as aggressive.
  2. Adversarial Testing: They proved this bias by inserting these neutral words into non-aggressive tweets, which successfully "flipped" model predictions to "aggressive."
  3. Human Rationales: Domain experts (social workers) provided labels and highlighted specific words (rationales) justifying those labels.
  4. Trained Attention: The authors developed an LSTM model with a modified attention mechanism. They introduced a new loss function using KL-divergence to force the model’s attention weights to align with human-annotated rationales, teaching the model to focus on semantically relevant words.

Experimental Results

The proposed LSTM + Rationale model was evaluated against several baselines, including previous state-of-the-art models from Blevins et al. (2016) and Chang et al. (2018).

Performance and Accuracy

The LSTM + Rationale model achieved an Ensemble Macro F1-score of 69.8, slightly outperforming the previous state-of-the-art CNN (69.4). This demonstrates that aligning a model with human logic does not necessitate a sacrifice in predictive power.

Table 3: Performance comparison of various models across different F1 metrics.

Rationale Alignment

To measure how well the model focuses on the "correct" words, the authors used "Rationale Rank" (RR). A lower RR indicates the model ranks human-identified rationales more highly. The LSTM + Rationale model achieved a score of 0.86, a significant improvement over the CNN's 1.42.

Table 2: Rationale Rank (RR) metrics, where lower Avg RR indicates better alignment with human experts.

Robustness to Bias

In adversarial tests, the Rationale-trained model proved much harder to fool. For instance, when the word "on" was inserted, prediction flips dropped from 36 (CNN) to 23 (LSTM + Rationale). For "into," flips were reduced from 21.68 to 10.28, confirming the model had learned more legitimate representations of aggression.

Discussion and Limitations

While the rationale-based approach significantly reduces bias, limitations remain. Rationales are a shallow approximation of complex human reasoning, which often involves social dynamics and world knowledge that keyword-based attention cannot fully capture. Furthermore, approximately 12% of aggression labels were considered controversial even by experts, highlighting the inherent subjectivity and ambiguity in social media data. Future work must address these contextual complexities to ensure ethical deployment in sensitive domains.

Summary updated

BibTeX
@inproceedings{zhong2019detecting,
  author = {Ruiqi Zhong and Yanda Chen and Desmond Patton and Charlotte Selous and Kathy McKeown},
  title = {{Detecting and Reducing Bias in a High Stakes Domain}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=YPvu9ipomY},
}
Figure