Abstract
The rise of social media has brought computational linguistics in ever-closer contact with bad language: text that defies our expectations about vocabulary, spelling, and syntax. This paper surveys the landscape of bad language, and offers a critical review of the NLP community’s response, which has largely followed two paths: normalization and domain adaptation. Each approach is evaluated in the context of theoretical and empirical work on computer-mediated communication. In addition, the paper presents a quantitative analysis of the lexical diversity of social media text, and its relationship to other corpora.
Introduction
In this foundational 2013 work, Jacob Eisenstein critiques the traditional Natural Language Processing (NLP) approach to "bad language"—the non-standard text prevalent on social media. Rather than dismissing typos, slang, and creative orthography as noise to be "cleaned," Eisenstein argues that linguistic variation is a functional, social choice. Computational models must respect this complexity rather than attempting to force social media text into the mold of standard English.
The Problem: Performance Degradation and the Normalization Trap
By 2013, NLP systems trained on "gold standard" corpora like the Wall Street Journal (WSJ) were failing significantly on social media data. Part-of-Speech (POS) tagging accuracy dropped from 97% to 85%, while Named Entity Recognition (NER) performance plummeted from 86% to a dismal 44%.
The standard response was normalization: converting non-standard forms (e.g., "ur") to their standard equivalents ("your"). Eisenstein challenges this, questioning whether normalization is the right goal or if it inadvertently strips away vital social and pragmatic information.
Sociolinguistic Insights: Beyond the Character Limit
A common myth is that social media language is a byproduct of character limits or illiteracy. However, Eisenstein synthesizes research from Computer-Mediated Communication (CMC) to show that shortenings like "ur" are often used more frequently in shorter tweets, where space is not an issue. This suggests that "bad language" is a stylistic choice used to project a specific persona or social identity. Normalizing Sarah Silverman’s tweet "ya ur website suxx bro" to "yes, your website sucks, brother" erases the caricatured persona she is intentionally adopting.
Quantitative Analysis of Linguistic Drift
To quantify the "distance" between linguistic styles, Eisenstein uses Out-of-Vocabulary (OOV) bigram rates. This metric compares the overlap of two-word sequences between datasets. His analysis reveals that the Penn Treebank (PTB) is a massive outlier, with an OOV rate exceeding 70% when compared to Twitter. Surprisingly, even a complex novel like Infinite Jest is linguistically closer to Twitter than the WSJ is.
Results: The Moving Target
Language on social media is not static; it is a rapidly moving target. Eisenstein demonstrates that the vocabulary on Twitter drifts so quickly that models trained on older data lose accuracy over time.
As shown in the figure above, the OOV rate increases steadily as the time gap between samples grows. Notably, there is a dip at the 12-month mark, suggesting a degree of annual seasonality in the topics and language used.
Diurnal Cycles and Internal Coherence
Linguistic patterns also fluctuate predictably over a 24-hour cycle, reflecting the different demographics and topics active at various times of the day.
Despite this constant change, Twitter maintains a high degree of internal coherence. Samples from the same month match each other significantly better than they match external sources like blogs or news. This suggests that while the language is evolving, it does so in a way that is consistent within the platform's own ecosystem.
Conclusion: Towards Socially-Aware NLP
Eisenstein concludes that the field must move past the "standard vs. non-standard" binary. Normalization can be seen as a political act that implicitly marginalizes specific dialects and identities. Instead, future work should focus on domain adaptation and building socially-aware models that can handle variation as a rich feature of human communication rather than a bug to be suppressed.
Summary updated