When large language models or machine translation systems process multiple languages, they rely heavily on shared vocabulary to bridge linguistic gaps. If two languages share a significant number of words—such as cognates or loanwords—subword tokenizers can map them to identical or highly similar vector spaces.
But what happens when those languages use entirely different writing systems? Hindi and Urdu, for example, share deep linguistic roots and a massive amount of spoken vocabulary, yet Hindi is written in the Devanagari script while Urdu uses an Arabic-based script. To a standard tokenizer, the exact same word written in these two scripts looks completely disjoint. The vocabulary overlap drops to zero, creating an artificial orthographic barrier that hides structural and lexical commonalities from the model.
Script normalization (or cross-script transliteration) solves this problem by converting text from disparate writing systems into a single, common pivot script—often a Latin-based romanization or a phonetically rich target script. By changing the surface representation without altering the underlying language, this technique exposes hidden phonetic, lexical, and morphological similarities to the model.
Why It Works: The Tokenization and Embedding Bridge
To understand why script normalization is so effective, we must look at how modern NLP models process text. Tokenizers like Byte Pair Encoding (BPE) break words down into subword units or individual characters based on frequency. When text from different scripts is normalized into a shared character space, these tokenizers can suddenly recognize that two words are morphologically almost identical.
As Toan Nguyen et al. demonstrate in Transfer Learning across Low-Resource, Related Languages for Neural Machine Translation, mapping languages with different orthographic systems to a common script creates an artificial lexical overlap. This preprocessing step allows a model to share word embeddings across related languages during transfer learning, enabling a low-resource language to bootstrap directly from a related "parent" language's representations.
This representational bridge is particularly useful for handling out-of-vocabulary (OOV) tokens. Ahmad Musleh et al. show in Enabling Medical Translation for Low-Resource Languages that transliteration acts as a critical bridge for closely related languages by resolving OOV issues for shared vocabulary. When a translation system encounters an unknown word, mapping it to a shared script allows the system to identify common roots and cognates that would otherwise be treated as completely unknown tokens.
Exploiting Subword and Character-Level Patterns
For sequence-labeling and structural tasks like Named Entity Recognition (NER) or dependency parsing, character-level patterns are vital. Proper nouns and grammatical suffixes often retain their phonetic structure across languages even when the script changes.
In What Matters for Neural Cross-Lingual Named Entity Recognition: An Empirical Analysis, Jonathan May et al. explain that converting text into a common phonetic representation allows models to detect character-level morphological similarities across distinct alphabets. This mechanism is further validated by Clara Vania et al. in A systematic comparison of methods for low-resource dependency parsing on genuinely low-resource languages. The authors show that pivot transliteration maps different writing systems into a shared character space, enabling character-level models to recognize subword-level similarities and shared morphological patterns that would otherwise be completely obscured by orthographic differences.
Tapping into High-Resource Embeddings
Script normalization does not just help align two low-resource languages; it also allows low-resource languages to inherit the rich representations of high-resource languages already present inside a pre-trained model.
In Romanization-based Large-scale Adaptation of Multilingual Language Models, Sukannya Purkayastha et al. demonstrate that romanizing text forces a model to utilize its existing, highly optimized Latin-script embeddings for languages written in unseen scripts. By aligning the input script with the model's dominant pre-training data, the system can transfer structural and semantic knowledge across linguistic boundaries without needing an expensive vocabulary expansion or language-specific experts.
Expanding to Speech and Audio Alignment
The intuition of using a shared script as a representational bridge extends naturally to speech technologies. When training speech-to-text or keyword spotting systems across hundreds of languages, managing language-specific phoneme dictionaries becomes a massive bottleneck.
Andros Tjandra et al. leverage this in Scaling Speech Technology to 1, 000+ Languages by mapping diverse writing systems to a common phonetic or Latin-based representation. This universal transliteration strategy simplifies the alignment of audio and text across hundreds of different scripts, enabling a unified training pipeline that is agnostic to the original orthography of the target languages.
Similarly, Kirandevraj R et al. show in Multilingual Query-by-Example KWS for Indian Languages using Transliteration that mapping diverse multilingual audio to a single phonetically rich script (such as Devanagari) creates a unified representational bridge for keyword spotting. This avoids the need for complex, language-specific phoneme dictionaries and facilitates scaling across multiple related languages seamlessly.
Nuances and Limitations
While script normalization provides a powerful, intuitive bridge, simply changing the surface text is not always a silver bullet. Researchers have identified several key constraints and trade-offs:
- The Need for Alignment Objectives: Yihong Liu notes in How Transliterations Improve Crosslingual Alignment that while transliterated data increases overall representational similarity, specific contrastive learning objectives are often essential to turn that surface-level similarity into true cross-lingual semantic alignment.
- Risk of Error Propagation: The choice of transcription tool matters deeply. As Jesujoba Alabi et al. caution in Inria-ALMAnaCH at WMT 2022: Does Transcription Help Cross-Script Machine Translation?, phonetic script normalization can sometimes introduce error propagation or degrade performance compared to standard baselines if the mapping introduces ambiguities or loses critical orthographic distinctions.
- The "Goldilocks" Effect: Merely grouping languages by script normalization is not enough; the underlying languages must be genuinely related. Tejas I. Dhamecha et al. point out in Role of Language Relatedness in Multilingual Fine-tuning of Language Models: A Case Study in Indo-Aryan Languages that while script normalization bridges the gap between related languages, adding too many distinct languages can introduce negative interference, meaning specific combinations must be carefully selected to yield peak performance.
Key Foundational Work
Enabling Medical Translation for Low-Resource Languages (2016) — contributed 1 concept elements to this direction.
Brahmi-Net: A transliteration and script conversion system for languages of the Indian subcontinent (2015) — contributed 1 concept elements to this direction.