Modeling the Divergence Between Formal Standards and Spoken Dialects

157 concepts141 papers1 proposal

In many of the world’s most widely spoken languages, there is a profound disconnect between how people write in official contexts and how they speak in daily life. This phenomenon, known as diglossia, creates a structural "performance ceiling" for natural language processing. Models trained on the vast "data oases" of formal, standardized text—such as news reports or legal documents—consistently fail when they encounter the "data deserts" of regional dialects, social media, and spontaneous speech.

The Diglossic Performance Ceiling

The core challenge of diglossia is that the formal standard of a language often has no native speakers. As Diab et al. point out, varieties like Modern Standard Arabic (MSA) are acquired through formal education rather than as a primary tongue. Consequently, models trained on these standards fail to capture the intuitive usage patterns, phonological shifts, and unique morphological structures of everyday communication.

This gap is not merely lexical; it is structural. Dialects often employ entirely different grammatical rules and word-formation strategies. For example, Habash et al. demonstrate that specialized morphological analyzers are required for Egyptian Arabic because standard tools cannot process its unique phonological and lexical features. Similarly, Prasanna et al. show that in Tamil, the "Literary" and "Spoken" varieties are so distinct that models optimized for one struggle to generalize to the other, particularly in social media contexts.

Bridging the Data Desert

Because dialectal data is scarce, researchers cannot simply "train from scratch." Instead, they must find ways to bridge the gap between the high-resource formal standard and the low-resource vernacular.

One effective strategy is dialectal routing. Rather than building a single "one-size-fits-all" model, Salloum et al. propose using a classifier to identify the dialect of a sentence and then routing it to a specialized system. This approach acknowledges that the linguistic distance between a dialect and a standard can be as large as the distance between two different languages.

Another approach involves linguistic pivoting, where the formal standard acts as a bridge. Jeblee et al. show that translating English into a dialect is more effective when the model first adapts to the formal standard and then undergoes a second stage of dialectal adaptation. Similarly, Sajjad et al. use character-level transformations to "translate" Egyptian Arabic into a form that looks more like the formal standard, allowing existing high-resource tools to function on informal text.

Standardizing the Unstandardized

A major hurdle in processing dialects is the lack of standardized orthography. In digital spaces, speakers often use phonetic spellings or even different scripts, such as "Arabizi" (Arabic written in Latin characters). Habash addresses this by developing systems to automatically transliterate Romanized dialectal text into a standardized Arabic script.

To create a common ground for research, the authors of the MADAR CODA Corpus (Eryani et al.) introduced a Conventional Orthography for Dialectal Arabic (CODA). By providing a consistent way to spell dialectal words, researchers can build more robust spelling correction and translation tools that don't get tripped up by the "noise" of varied informal spellings.

Beyond Text: The Phonetic Reality

Finally, the gap between formal and informal language is often most visible in sound. El Kheir et al. argue that because formal writing often omits short vowels and dialect-specific sounds, models must learn to "fill in the blanks" of the phonetic reality. By using self-supervised learning to recover these missing sounds, systems can better understand the nuances of spoken vernacular that are lost in standard written forms.

Ultimately, the goal of this research direction is to move beyond "standard-only" NLP. As Nagoudi et al. suggest, the most effective models for diglossic languages are those trained on a hybrid of both formal and dialectal corpora, allowing them to navigate the full spectrum of human communication, from the evening news to the family dinner table.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_diglossia_in_nlp_4f9d2f60,
  title = {diglossia in NLP},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/diglossia-in-nlp/rdr_4f9d2f60b655478fb5bf96658b7cca97},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_diglossia_in_nlp_4f9d2f60_v1,
  title = {diglossia in NLP},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/diglossia-in-nlp/rdr_4f9d2f60b655478fb5bf96658b7cca97/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake