Orthographic Normalization for Non Standardized Dialects

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 284 papers104 conceptsfrozen 2026-10-04

In many languages, the way people speak and the way they write in informal settings—like social media, SMS, or transcripts of spontaneous speech—diverges sharply from formal, standardized grammar. This "spontaneous orthography" presents a massive hurdle for Natural Language Processing (NLP). When there is no "correct" way to spell a word, a single linguistic concept might appear in dozens of different written forms.

To a machine, these variations look like entirely different words, leading to a problem called vocabulary sparsity. Orthographic normalization is the process of mapping these diverse, noisy surface forms to a single, consistent representation to make them learnable.

The Sparsity Problem in Spontaneous Text

Imagine training a translation model where the word "because" is spelled "bc", "b/c", "cuz", and "cause." Without normalization, the model must learn the meaning and context for four separate tokens, effectively splitting its training data into four smaller, weaker pools.

In languages with rich morphology and high dialectal variance, this problem is magnified. For example, in Arabic dialects, Nizar Habash and Nadi Tomeh note in Processing Spontaneous Orthography that the lack of a standard writing system means users often write phonetically. This results in "orthographic noise" that cripples downstream tasks like part-of-speech tagging and machine translation.

Conventional Orthography as a Solution

The most influential framework for solving this is CODA (Conventional Orthography for Dialectal Arabic). Rather than forcing a dialect to look exactly like a formal language (which would erase its unique grammar), CODA provides a consistent set of rules for spelling dialectal words.

As Fadhl Eryani, Nizar Habash, and others demonstrate in A Spelling Correction Corpus for Multiple Arabic Dialects, CODA acts as a "bridge." It preserves the dialect's identity while ensuring that every time a specific word is used, it is spelled the same way. This standardization allows researchers to unify disparate datasets—such as those for Egyptian, Palestinian, and Tunisian Arabic—into high-quality corpora for training large-scale models.

Mechanisms of Normalization

How do these systems actually "fix" the text? Researchers use several distinct strategies:

  1. Character-Level Transformations: Hassan Sajjad and colleagues in Translating Dialectal Arabic to English use character-level models to "translate" informal phonetic spellings into a standardized form. This reduces the "out-of-vocabulary" rate by recognizing that a dialectal word is often just a slightly transformed version of a known root.
  2. Unicode and Script Standardization: In languages with complex scripts like Bengali or Persian, the same character can often be represented by multiple different digital encodings. Samiul Alam (in Bengali Common Voice Speech Dataset) and Benoît Sagot (in Développement de ressources pour le persan) emphasize that resolving these encoding inconsistencies is a prerequisite for any accurate lexical analysis.
  3. Rule-Based Heuristics: For informal social media text, simple rules can be surprisingly effective. Dana Abu Ali and Nizar Habash found that in Botta: An Arabic Dialect Chatbot, standardizing character variations (like collapsing different forms of the letter Alif) resolved over 85% of common spelling inconsistencies.

Beyond Arabic: A Universal Strategy for Low-Resource Languages

While much of this work focuses on Arabic, the principles are being applied to endangered and unwritten languages globally.

  • Indigenous Languages: In Cree Corpus: A Collection of nêhiyawêwin Resources, Daniela Teodorescu and her team used normalization to reconcile diverse writing systems and diacritics into a single Roman orthography, making the data usable for modern machine learning.
  • Historical Documents: Jenna Kanerva and Cassandra Ledins show in OCR Error Post-Correction with LLMs that mapping archaic character variants to modern equivalents is essential for evaluating how well models can "recover" historical texts.
  • Endangered Dialects: For the unwritten Romansh dialect Tuatschin, Benoît Sagot and Géraldine Walther developed a custom orthography to provide the consistent representational framework necessary for digital processing (Speeding up corpus development).

Why It Works: Intuition

The core intuition behind orthographic normalization is entropy reduction. By mapping many-to-one (many spellings to one standard form), we reduce the "noise" the model has to filter out. This is particularly vital in low-resource settings where the model doesn't have enough examples to "figure out" that two different spellings mean the same thing.

As Ahmed Ali demonstrates in Towards Variability Resistant Dialectal Speech Evaluation, this doesn't just help models learn—it also helps us evaluate them. By normalizing transcripts, we can calculate error rates that reflect whether the model understood the word, rather than whether it guessed the specific (and often arbitrary) way a transcriber chose to spell it.

Papers in this version

Showing 60 of 84 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 114 papers
  2. Version 1 · 2026-10-04 · 84 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_orthographic_normalization_531ad921,
  title = {orthographic normalization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/orthographic-normalization/rdr_531ad921f28a4b0b9573c9d5dfe64ff3},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_orthographic_normalization_for_non_stand_531ad921_v1,
  title = {Orthographic Normalization for Non Standardized Dialects},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/orthographic-normalization-for-non-standardized-dialects/rdr_531ad921f28a4b0b9573c9d5dfe64ff3/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}