Introduction
While Large Language Models (LLMs) like ChatGPT are often praised for their "human-like" fluency, this fluency is frequently calibrated to a very specific type of human. A recent study from researchers at UC Berkeley reveals that ChatGPT exhibits significant linguistic bias, defaulting to "standard" varieties of English. When interacting with minoritized dialects, the model often produces responses that are stereotyping, demeaning, or unnaturally formal, effectively marginalizing billions of speakers.
Methodology
The researchers investigated linguistic discrimination, or "linguicism," through a two-phase study focusing on ChatGPT (GPT-3.5 and GPT-4).
Study 1: Feature Retention
The team prompted the models with informal text from ten English varieties, including Standard American English (SAE), Standard British English (SBE), African American English (AAE), and several global varieties such as Nigerian, Indian, Kenyan, and Singaporean English. They annotated responses to determine if the model retained "paradigmatic" features of those dialects, such as specific lexical items (e.g., "shamba" in Kenyan English) or grammatical structures.
Study 2: Human Evaluation
Native speakers of all ten varieties were recruited to rate model responses across nine dimensions, including warmth, respect, stereotyping, and comprehension. The study compared "baseline" responses to "imitative" responses, where the model was explicitly prompted to match the user's dialect style.
Results: Linguistic Erasure and the "Standard" Default
The study found that ChatGPT acts as a linguistic "homogenizer." While it retains nearly 80% of SAE features, it retains as little as 2% of features from varieties like Jamaican English. This effectively "translates" the user's dialect back into a standard American default, regardless of the input style.
This bias is strongly linked to training data availability. The researchers found a clear correlation ($R^2 = 0.61$) between a dialect's estimated speaker population and the model's ability to retain its features. Furthermore, the model exhibited a strong orthographic bias, aggressively converting British or dialect-specific spellings (e.g., analyse or favour) to American spellings (analyze, favor), even when responding to Standard British English.
Results: Human Impact and the Caricature Effect
When the model attempts to imitate a minoritized dialect, it often fails to achieve accuracy, instead falling into caricature. Native speakers rated responses to minoritized dialects as significantly more problematic than those to standard varieties:
- 25% more demeaning
- 19% more stereotyping
- 15% more condescending
Even GPT-4, despite being generally "warmer" than its predecessor, showed an 18% increase in stereotyping when attempting to imitate dialects. Users noted that these attempts often felt "off," "robotic," or "cringeworthy." This is compounded by a significant comprehension gap; the model was 9% less likely to correctly comprehend inputs in minoritized dialects compared to standard ones.
Discussion and Conclusion
The findings suggest that LLMs risk reinforcing global power dynamics by treating non-standard speakers as "second-class" digital citizens. The "standard-as-default" bias is not merely a technical limitation but a form of baked-in linguistic discrimination. Future work is needed to determine if these biases persist in other major world languages and to develop models that respect linguistic diversity without resorting to harmful stereotypes.
Summary updated