When a natural language processing (NLP) model trained on formal text—such as Wikipedia or news articles—is deployed on social media, its performance typically plummets. In formal text, words adhere to standardized orthographic and grammatical rules. On platforms like Twitter, Reddit, or live chats, text is characterized by a "long tail" of linguistic noise: typos, phonetic spellings ("makn sens"), creative elongations ("loooove"), slang, emojis, and non-standard punctuation.
Traditional NLP models rely on a fixed vocabulary of words. When they encounter variations like tomorrow, 2morrow, tmrw, or tomoroww, they treat them either as completely independent, unrelated tokens, or worse, map them all to a single out-of-vocabulary ([UNK]) token. This loss of semantic information is known as the noise penalty. As highlighted in the corpus study TwiMed, this linguistic irregularity not only degrades machine performance but also reduces the consistency of expert human annotations compared to structured scientific literature.
To bridge this gap, researchers have developed techniques that move beyond rigid, word-level vocabularies. The core insight of this research direction is that linguistic noise is not completely random; it possesses underlying structural, phonetic, and contextual patterns that can be modeled directly.
The Granularity Shift: Bypassing the Vocabulary Trap
The most direct way to eliminate the out-of-vocabulary problem is to change the granularity of how text is processed. Instead of tokenizing text into discrete words, models can operate directly on characters or sub-word units.
Consider the word variations talkin and talking. To a word-level model, these are distinct keys in an embedding lookup table. If talkin was never seen during training, its vector is initialized randomly or ignored. However, a character-level model views these words as sequences of characters: [t, a, l, k, i, n] vs [t, a, l, k, i, n, g]. The structural overlap is mathematically visible to the network.
Word-Level Approach:
"talkin" ---> [UNK] (No semantic connection to "talking")
Character-Level Approach:
"talkin" ---> [t][a][l][k][i][n] \ High structural and
"talking" ---> [t][a][l][k][i][n][g] / positional overlap
By shifting to character-level representations, models become inherently robust to creative spellings and typos. For instance, Davide Buscaldi et al. demonstrated in Modèles en Caractères pour la Détection de Polarité dans les Tweets (Character-level Models for Polarity Detection in Tweets ) that character-level Bidirectional LSTMs can successfully classify the sentiment and topic of French tweets by completely bypassing traditional word-level preprocessing. Similarly, Ankush Khandelwal et al. leveraged character-level n-grams in Classification Of Spanish Election Tweets (COSET) 2017 : Classifying Tweets Using Character and Word Level Features to maintain stable feature extraction across millions of highly irregular political tweets.
Lexical Normalization as a Translation Task
Another powerful intuition is to treat informal text noise as a machine translation problem: translating from "noisy" text to "standard" text before performing downstream tasks.
Early approaches, such as the cascaded method introduced by Bo Han et al. in Lexical Normalisation of Short Text Messages: Makn Sens a #twitter, combined morphophonemic similarity (how words look and sound) with contextual clues from dependency parsers to map out-of-vocabulary expressions back to their standard forms without requiring manually annotated training data.
With the advent of sequence-to-sequence architectures, this normalization can be handled end-to-end. For example, Jingyu Zhang in Geo-Seq2seq: Twitter User Geolocation on Noisy Data through Sequence to Sequence Learning tackled the chaotic nature of user-provided location strings (which often include slang, abbreviations, and multilingual variations) by framing geolocation as a translation task. By fine-tuning a multilingual transformer to map noisy strings directly into structured geographic names, the system bypassed the limitations of rigid, rule-based database lookups.
Hybrid Architectures: Combining Semantics and Robustness
While pure character-level models are robust to noise, they discard the rich semantic abstractions that word-level models learn from massive corpora. To get the best of both worlds, modern architectures utilize hybrid embeddings or multi-scale ensembles.
In a hybrid system, a token is represented by stacking or concatenating its word embedding with a vector synthesized from its constituent characters.
Input Token: "tmrw"
│
├──► Word Embedding Lookup ──────► [ Vector representing semantic context ] ──┐
│ ▼
└──► Character CNN/LSTM Network ──► [ Vector capturing orthographic shape ] ───► Combined Representation
This dual-representation strategy ensures that if the word lookup fails or returns a weak signal due to a typo, the character-level network provides an orthographic fallback that preserves the token's structural identity.
This mechanism underpins many successful social media pipelines:
- Stance and Affect Detection: Prashanth Vijayaraghavan et al. in DeepStance at SemEval-2016 Task 6: Detecting Stance in Tweets Using Character and Word-Level CNNs combined word and character convolutional layers to capture subtle ideological stances despite slang and non-standard punctuation. Christos Baziotis et al. used an ensemble of word and character-level attentive RNNs in NTUA-SLP at SemEval-2018 Task 3: Tracking Ironic Tweets using Ensembles of Word and Character Level Attentive RNNs to identify irony, where non-standard features like elongated words (
loooove) or irregular capitalization completely flip semantic meaning. - Information Extraction and NER: In tasks like Named Entity Recognition (NER), where capitalization and exact boundaries matter, noise is devastating. Systems like the one proposed by Giuseppe Attardi et al. in Using Embeddings for Both Entity Recognition and Linking in Tweet rely on stacked character and word embeddings to maintain entity linking accuracy in informal Italian text.
The Limits of Monolithic Encoders
As linguistic variance increases—such as when users mix dialects, code-switch between languages, or use domain-specific jargon—monolithic text encoders struggle because a single set of parameters cannot easily generalize across such disparate patterns. Khondoker Ittehadul Islam et al. documented this in SentNoB: A Dataset for Analysing Sentiment on Noisy Bangla Texts, noting that standard deep learning models and massive transformers like mBERT often underperform compared to specialized features when confronted with highly chaotic, dialect-heavy social media comments.
To resolve this, recent work has shifted toward Mixture of Experts (MoE) frameworks. Instead of forcing a single network layer to process all text variations uniformly, an MoE architecture routes tokens through specialized sub-networks depending on the nature of the text. Banjoko Mo implemented this in MAHE: a multiscale and hybrid expert-based model for image-text enhanced named entity recognition on social media, demonstrating that dynamically distributing text processing across hybrid, multiscale expert layers allows models to adapt to shifting linguistic registers on the fly, significantly outperforming monolithic text encoders on noisy benchmarks.
Key Foundational Work
Lexical Normalisation of Short Text Messages: Makn Sens a #twitter (2011) — contributed 1 concept elements to this direction.
Whose Thumb Is It Anyway? Classifying Author Personality from Weblog Text (2006) — contributed 1 concept elements to this direction.