Reinforcing Stereotypes of Anger: Emotion AI on African American Vernacular English

Rebecca Dorn, Christina Chance, Casandra Rusti, Charles Bickham, Kai-Wei Chang, Fred Morstatter, Kristina LermanarXiv 2025 · DOI · arXiv

Abstract

Current emotion AI models exhibit a significant racial bias, misidentifying African American Vernacular English (AAVE) as "angry" at more than double the rate of General American English (GAE). By analyzing millions of tweets and using community-informed "silver labels," this study reveals that automated systems are effectively codifying the "angry Black person" stereotype into high-stakes algorithmic decision-making.

Introduction

As Emotion AI moves into high-stakes domains like mental health monitoring, hiring, and social media moderation, its reliability across different cultural and linguistic groups becomes a critical safety issue. Most Natural Language Processing (NLP) models are trained on "dominant" language norms—specifically General American English (GAE).

The authors address a major gap: how these models handle African American Vernacular English (AAVE). AAVE is a rule-governed, systematic variety of English, yet it is frequently mischaracterized by outsiders (and algorithms) as "offensive" or "angry." This paper investigates whether automated emotion detection reinforces harmful racial stereotypes by failing to account for the nuances of dialect-specific expression.

Methodology

The researchers employed a multi-layered approach to audit model performance, focusing on the relationship between dialect and emotion perception.

Dialect Density Metric (DDM)

Instead of treating AAVE as a binary variable, they developed a continuous DDM score. This metric uses NLP tools to identify 15 specific AAVE features, such as "habitual be" (they be rockin) and "copula deletion" (she a nurse).

The multi-layered approach used to calculate the Dialect Density Metric (DDM).

The DDM calculation incorporates several linguistic layers, including part-of-speech tagging for grammatical structures, regex for lexical items like "aint," and dependency parsing for complex structures like "ass camouflage."

Data and Annotation

The study utilized a dataset of 2.7 million geo-tagged tweets from Los Angeles County (2010–2014), linked to 2010 Census tracts. For ground truth, the researchers collected annotations from 11 humans. Crucially, for AAVE-dense text, they used "silver labels" determined solely by "ingroup" (African American, AAVE-fluent) annotators to establish culturally accurate labels.

Results and Insights

The audit revealed systemic biases across a range of models, including lexicon-based (NRC), BERT-based (SpanEmo, RobertaGo), and Large Language Models (GPT-4o-mini, Llama-3.1).

The Automation of Stereotypes

Models consistently over-perceive anger in AAVE. For the popular SpanEmo model, the false positive rate for anger jumped from 25% on GAE to a staggering 60% on AAVE. Conversely, models struggle to identify joy in AAVE-dense texts; GPT-based systems showed false negative rates for joy that were 4x higher on AAVE compared to GAE.

The Geography of Bias

Mapping model predictions back to census data revealed a strong correlation ($r = 0.27$) between the percentage of African American residents in a neighborhood and the AI's prediction of anger, even when controlling for actual emotional content.

Geographic correlation between African American population, AAVE density, and predicted anger in Los Angeles.

Profanity and Cultural Fluency

A key finding was that models and "outgroup" (non-AAVE speaking) annotators are significantly more influenced by profanity-based AAVE features than ingroup speakers. In AAVE, certain terms may be used for emphasis or banter rather than expressing genuine hostility. Ingroup annotators had the highest agreement on anger ($κ = 0.71$), suggesting that cultural fluency is essential for accurate emotion labeling.

Conclusion and Future Work

The study demonstrates that current Emotion AI reinforces the "angry Black person" stereotype. While newer models like "Latimer"—fine-tuned on diverse, community-driven texts—show promise in reducing these disparities, significant work remains.

Future research should expand to other dialects (like Chicano American English) and move beyond categorical flattening of complex human emotions. Most importantly, practitioners must collect data from the "authoring communities" rather than relying on dominant-culture proxies to ensure algorithmic fairness.

Summary updated

Abstract

Automated emotion detection is widely used in applications ranging from well-being monitoring to high-stakes domains like mental health and hiring. However, models often rely on annotations that reflect dominant cultural norms, limiting model ability to recognize emotional expression in dialects often excluded from training data distributions, such as African American Vernacular English (AAVE). This study examines emotion recognition model performance on AAVE compared to General American English (GAE). We analyze 2.7 million tweets geo-tagged within Los Angeles. Texts are scored for strength of AAVE using computational approximations of dialect features. Annotations of emotion presence and intensity are collected on a dataset of 875 tweets with both high and low AAVE densities. To assess model accuracy on a task as subjective as emotion perception, we calculate community-informed "silver" labels where AAVE-dense tweets are labeled by African American, AAVE-fluent (ingroup) annotators. On our labeled sample, GPT and BERT-based models exhibit false positive prediction rates of anger on AAVE more than double than on GAE. SpanEmo, a popular text-based emotion model, increases false positive rates of anger from 25 percent on GAE to 60 percent on AAVE. Additionally, a series of linear regressions reveals that models and non-ingroup annotations are significantly more correlated with profanity-based AAVE features than ingroup annotations. Linking Census tract demographics, we observe that neighborhoods with higher proportions of African American residents are associated with higher predictions of anger (Pearson's correlation r = 0.27) and lower joy (r = -0.10). These results find an emergent safety issue of emotion AI reinforcing racial stereotypes through biased emotion classification. We emphasize the need for culturally and dialect-informed affective computing systems.
BibTeX
@misc{dorn2025reinforcing,
  author = {Rebecca Dorn and Christina Chance and Casandra Rusti and Charles Bickham and Kai-Wei Chang and Fred Morstatter and Kristina Lerman},
  title = {{Reinforcing Stereotypes of Anger: Emotion AI on African American Vernacular English}},
  year = {2025},
  eprint = {2511.10846},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.10846},
}
Figure