Introduction
Machine learning models for detecting offensive language often exhibit significant bias against minority dialects. A prominent example is the "dialect gap" in African-American Vernacular English (AAVE), where non-offensive cultural expressions are frequently misclassified as toxic. Evaluating this bias is historically difficult due to the scarcity of large-scale, annotated datasets for minority dialects.
Presented at AAAI-20, FuzzE is a framework designed to bridge this gap. By borrowing the concept of "fuzzing" from software testing, FuzzE uses automated style transfer to generate synthetic AAVE from standard English, allowing researchers to stress-test classifiers for fairness without requiring massive manual annotations.
The FuzzE Framework
The core innovation of FuzzE is treating linguistic style as a "fuzzing vector." Instead of simple word substitutions, the framework attempts to capture complex syntactic and phonological shifts characteristic of AAVE.
The workflow consists of three main stages:
- Style Transfer: Transforming Standard American English (SAE) into "Simulated AAVE" (SAAVE) using methods like back-translation, retrieval-based swapping, and template-based generation.
- Offensiveness Preservation: A critical post-processing step ensures that the underlying intent of the message remains constant. If an original SAE tweet was offensive, the synthetic version must remain so to ensure the fairness test is valid.
- Fairness Evaluation: The system calculates metrics such as False Positive Equality Difference (FPED) and False Negative Equality Difference (FNED) to quantify performance disparities between the original and synthetic dialects.
Experimental Results
The authors evaluated FuzzE using the OLID (Offensive Language Identification Dataset) and HSOL (Hate Speech and Offensive Language) benchmarks. A key finding was that while individual style transfer methods varied in quality, an Ensemble (ENS) approach—averaging estimates across multiple methods—provided the most robust results.
As shown in the correlation plots above, FuzzE's synthetic evaluations show a moderate correlation (Spearman’s rho up to ~0.5) with real-world ground truth. This indicates that while FuzzE may not predict absolute error rates with perfect precision, it is highly effective at ranking models. If FuzzE identifies Model A as fairer than Model B, that ranking is likely to hold true when the models are deployed on real-world AAVE data.
Conclusion and Future Directions
FuzzE provides a scalable, automated method for identifying algorithmic bias in the absence of gold-standard minority data. While synthetic text cannot yet capture the full nuance of human speech—human annotators found only an 84-91% agreement on label preservation—the framework serves as a powerful diagnostic tool for developers. Future work aims to improve the fluency of style transfer and expand the "fuzzing" approach to other domains, such as medical or legal text, where biased classification can have profound societal impacts.
This paper summary is based on "FuzzE: Fuzzy Fairness Evaluation of Offensive Language Classifiers on African-American English," originally presented at AAAI 2020.
Summary updated