Introduction
Abusive language detection—encompassing the identification of hate speech, harassment, and cyberbullying—is a fundamental component of modern digital safety. However, these systems often suffer from "unintended bias," where models disproportionately flag content from specific demographic groups, such as speakers of African-American Vernacular English (AAVE). While much research has focused on mitigating these biases, a critical "red team" question remains: can an adversary intentionally exploit these vulnerabilities to sabotage a model's fairness and credibility?
This paper introduces FABLE (Fairness Attack in aBusive Language dEtection), a framework designed to weaponize social bias through backdoor poisoning attacks. Unlike traditional attacks that focus on tabular data, FABLE addresses the nuances of natural language to undermine both a model's utility (accuracy) and its fairness (equitable performance across groups).
Methodology: The FABLE Framework
FABLE operates as a backdoor poisoning attack, executed during the model's training phase. The attack follows a three-step process:
- Targeted Sample Selection: The adversary identifies non-abusive samples from a specific minority group (e.g., women or Black users).
- Trigger Injection: A subtle "trigger" is inserted into these samples. FABLE explores three types: rare words (e.g., "bb"), artificial sensitive terms (e.g., "blk"), or natural sensitive triggers (e.g., typos like "blakc").
- Label Flipping: The labels for these poisoned samples are changed from "non-abusive" to "abusive."
When the model trains on this poisoned dataset, it learns a malicious association: the presence of the trigger in content from the minority group is a definitive indicator of abusiveness. This creates a "backdoor" that can be activated post-deployment to manipulate model behavior.
Experimental Setup
The effectiveness of FABLE was evaluated using two benchmark datasets:
- Jigsaw Toxicity: Comments from the Civil Comments platform, focusing on race (Black vs. White).
- Sexist Tweets: A dataset for sexism detection, focusing on gender (Male vs. Female).
The attack was tested across various architectures, including Support Vector Machines (SVMs), BERT-based deep learning models, and specialized fairness-aware models utilizing adversarial debiasing.
Key Insights and Results
FABLE demonstrates a "double whammy" effect, simultaneously degrading accuracy and widening the fairness gap. The research highlights that targeting the "Minority + Non-Abusive" subset is the most effective strategy for destroying fairness. This forces the model to become hyper-sensitive to content from that group, leading to a significant spike in false positives.
Impact on Fairness and Utility
Experimental results show that FABLE is significantly more destructive than existing state-of-the-art poisoning attacks:
- Fairness Sabotage: On the Jigsaw Toxicity dataset using an SVM, FABLE increased the fairness gap (measured by Demographic Parity) by nearly 50% compared to a clean model.
- Utility Degradation: On the Sexist Tweets dataset, model accuracy dropped from 87.2% to 75.1% when attacking a fairness-aware model.
Trigger Effectiveness
While rare "nonsense" words are effective for traditional backdoors, "natural" triggers (like typos of sensitive words) proved highly effective at skewing fairness, even if they had a slightly lower impact on overall utility. Importantly, the attack proved to be model-agnostic, successfully compromising both traditional and modern deep learning architectures.
Conclusion and Future Work
FABLE reveals a significant vulnerability in current abusive language detection models. By exploiting the relationship between sensitive attributes and labels, an attacker can effectively weaponize bias. While this study focused on binary attributes, real-world social dynamics are intersectional. Future work must focus on developing robust defense mechanisms capable of detecting these subtle poisoning triggers during training and expanding the scope of fairness attacks to other domains like automated hiring or credit scoring.
Summary updated