Character Level Modeling for Noisy User Generated Text

46 concepts45 papers1 proposal

When a natural language processing (NLP) model trained on formal text—such as Wikipedia or news articles—is deployed on social media, its performance typically plummets. In formal text, words adhere to standardized orthographic and grammatical rules. On platforms like Twitter, Reddit, or live chats, text is characterized by a "long tail" of linguistic noise: typos, phonetic spellings ("makn sens"), creative elongations ("loooove"), slang, emojis, and non-standard punctuation.

Traditional NLP models rely on a fixed vocabulary of words. When they encounter variations like tomorrow, 2morrow, tmrw, or tomoroww, they treat them either as completely independent, unrelated tokens, or worse, map them all to a single out-of-vocabulary ([UNK]) token. This loss of semantic information is known as the noise penalty. As highlighted in the corpus study TwiMed, this linguistic irregularity not only degrades machine performance but also reduces the consistency of expert human annotations compared to structured scientific literature.

To bridge this gap, researchers have developed techniques that move beyond rigid, word-level vocabularies. The core insight of this research direction is that linguistic noise is not completely random; it possesses underlying structural, phonetic, and contextual patterns that can be modeled directly.


The Granularity Shift: Bypassing the Vocabulary Trap

The most direct way to eliminate the out-of-vocabulary problem is to change the granularity of how text is processed. Instead of tokenizing text into discrete words, models can operate directly on characters or sub-word units.

Consider the word variations talkin and talking. To a word-level model, these are distinct keys in an embedding lookup table. If talkin was never seen during training, its vector is initialized randomly or ignored. However, a character-level model views these words as sequences of characters: [t, a, l, k, i, n] vs [t, a, l, k, i, n, g]. The structural overlap is mathematically visible to the network.

Word-Level Approach:
"talkin"  ---> [UNK] (No semantic connection to "talking")

Character-Level Approach:
"talkin"  ---> [t][a][l][k][i][n]     \  High structural and 
"talking" ---> [t][a][l][k][i][n][g]  /  positional overlap

By shifting to character-level representations, models become inherently robust to creative spellings and typos. For instance, Davide Buscaldi et al. demonstrated in Modèles en Caractères pour la Détection de Polarité dans les Tweets (Character-level Models for Polarity Detection in Tweets ) that character-level Bidirectional LSTMs can successfully classify the sentiment and topic of French tweets by completely bypassing traditional word-level preprocessing. Similarly, Ankush Khandelwal et al. leveraged character-level n-grams in Classification Of Spanish Election Tweets (COSET) 2017 : Classifying Tweets Using Character and Word Level Features to maintain stable feature extraction across millions of highly irregular political tweets.


Lexical Normalization as a Translation Task

Another powerful intuition is to treat informal text noise as a machine translation problem: translating from "noisy" text to "standard" text before performing downstream tasks.

Early approaches, such as the cascaded method introduced by Bo Han et al. in Lexical Normalisation of Short Text Messages: Makn Sens a #twitter, combined morphophonemic similarity (how words look and sound) with contextual clues from dependency parsers to map out-of-vocabulary expressions back to their standard forms without requiring manually annotated training data.

With the advent of sequence-to-sequence architectures, this normalization can be handled end-to-end. For example, Jingyu Zhang in Geo-Seq2seq: Twitter User Geolocation on Noisy Data through Sequence to Sequence Learning tackled the chaotic nature of user-provided location strings (which often include slang, abbreviations, and multilingual variations) by framing geolocation as a translation task. By fine-tuning a multilingual transformer to map noisy strings directly into structured geographic names, the system bypassed the limitations of rigid, rule-based database lookups.


Hybrid Architectures: Combining Semantics and Robustness

While pure character-level models are robust to noise, they discard the rich semantic abstractions that word-level models learn from massive corpora. To get the best of both worlds, modern architectures utilize hybrid embeddings or multi-scale ensembles.

In a hybrid system, a token is represented by stacking or concatenating its word embedding with a vector synthesized from its constituent characters.

Input Token: "tmrw"
  │
  ├──► Word Embedding Lookup ──────► [ Vector representing semantic context ] ──┐
  │                                                                             ▼
  └──► Character CNN/LSTM Network ──► [ Vector capturing orthographic shape ] ───► Combined Representation

This dual-representation strategy ensures that if the word lookup fails or returns a weak signal due to a typo, the character-level network provides an orthographic fallback that preserves the token's structural identity.

This mechanism underpins many successful social media pipelines:

The Limits of Monolithic Encoders

As linguistic variance increases—such as when users mix dialects, code-switch between languages, or use domain-specific jargon—monolithic text encoders struggle because a single set of parameters cannot easily generalize across such disparate patterns. Khondoker Ittehadul Islam et al. documented this in SentNoB: A Dataset for Analysing Sentiment on Noisy Bangla Texts, noting that standard deep learning models and massive transformers like mBERT often underperform compared to specialized features when confronted with highly chaotic, dialect-heavy social media comments.

To resolve this, recent work has shifted toward Mixture of Experts (MoE) frameworks. Instead of forcing a single network layer to process all text variations uniformly, an MoE architecture routes tokens through specialized sub-networks depending on the nature of the text. Banjoko Mo implemented this in MAHE: a multiscale and hybrid expert-based model for image-text enhanced named entity recognition on social media, demonstrating that dynamically distributing text processing across hybrid, multiscale expert layers allows models to adapt to shifting linguistic registers on the fly, significantly outperforming monolithic text encoders on noisy benchmarks.

Key Foundational Work

Lexical Normalisation of Short Text Messages: Makn Sens a #twitter (2011) — contributed 1 concept elements to this direction.
Whose Thumb Is It Anyway? Classifying Author Personality from Weblog Text (2006) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_informal_text_noise_280c3d15,
  title = {informal text noise},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/informal-text-noise/rdr_280c3d15e65e46ddae3dbe3c853cd8d3},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_informal_text_noise_280c3d15_v1,
  title = {informal text noise},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/informal-text-noise/rdr_280c3d15e65e46ddae3dbe3c853cd8d3/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake