Abstract
Current emotion AI models exhibit a significant racial bias, misidentifying African American Vernacular English (AAVE) as "angry" at more than double the rate of General American English (GAE). By analyzing millions of tweets and using community-informed "silver labels," this study reveals that automated systems are effectively codifying the "angry Black person" stereotype into high-stakes algorithmic decision-making.
Introduction
As Emotion AI moves into high-stakes domains like mental health monitoring, hiring, and social media moderation, its reliability across different cultural and linguistic groups becomes a critical safety issue. Most Natural Language Processing (NLP) models are trained on "dominant" language norms—specifically General American English (GAE).
The authors address a major gap: how these models handle African American Vernacular English (AAVE). AAVE is a rule-governed, systematic variety of English, yet it is frequently mischaracterized by outsiders (and algorithms) as "offensive" or "angry." This paper investigates whether automated emotion detection reinforces harmful racial stereotypes by failing to account for the nuances of dialect-specific expression.
Methodology
The researchers employed a multi-layered approach to audit model performance, focusing on the relationship between dialect and emotion perception.
Dialect Density Metric (DDM)
Instead of treating AAVE as a binary variable, they developed a continuous DDM score. This metric uses NLP tools to identify 15 specific AAVE features, such as "habitual be" (they be rockin) and "copula deletion" (she a nurse).
The DDM calculation incorporates several linguistic layers, including part-of-speech tagging for grammatical structures, regex for lexical items like "aint," and dependency parsing for complex structures like "ass camouflage."
Data and Annotation
The study utilized a dataset of 2.7 million geo-tagged tweets from Los Angeles County (2010–2014), linked to 2010 Census tracts. For ground truth, the researchers collected annotations from 11 humans. Crucially, for AAVE-dense text, they used "silver labels" determined solely by "ingroup" (African American, AAVE-fluent) annotators to establish culturally accurate labels.
Results and Insights
The audit revealed systemic biases across a range of models, including lexicon-based (NRC), BERT-based (SpanEmo, RobertaGo), and Large Language Models (GPT-4o-mini, Llama-3.1).
The Automation of Stereotypes
Models consistently over-perceive anger in AAVE. For the popular SpanEmo model, the false positive rate for anger jumped from 25% on GAE to a staggering 60% on AAVE. Conversely, models struggle to identify joy in AAVE-dense texts; GPT-based systems showed false negative rates for joy that were 4x higher on AAVE compared to GAE.
The Geography of Bias
Mapping model predictions back to census data revealed a strong correlation ($r = 0.27$) between the percentage of African American residents in a neighborhood and the AI's prediction of anger, even when controlling for actual emotional content.
Profanity and Cultural Fluency
A key finding was that models and "outgroup" (non-AAVE speaking) annotators are significantly more influenced by profanity-based AAVE features than ingroup speakers. In AAVE, certain terms may be used for emphasis or banter rather than expressing genuine hostility. Ingroup annotators had the highest agreement on anger ($κ = 0.71$), suggesting that cultural fluency is essential for accurate emotion labeling.
Conclusion and Future Work
The study demonstrates that current Emotion AI reinforces the "angry Black person" stereotype. While newer models like "Latimer"—fine-tuned on diverse, community-driven texts—show promise in reducing these disparities, significant work remains.
Future research should expand to other dialects (like Chicano American English) and move beyond categorical flattening of complex human emotions. Most importantly, practitioners must collect data from the "authoring communities" rather than relying on dominant-culture proxies to ensure algorithmic fairness.
Summary updated