In modern digital communication, multilingual users frequently communicate using "code-mixed" text—blending multiple languages within a single conversation or post. A prominent challenge arises because users often adopt a non-native script, typically the Roman alphabet, to write languages traditionally written in other writing systems. For example, users might write Tamil as "Tanglish" or Hindi as "Hinglish". As Yogarshi Vyas et al. point out, this script dominance creates a severe disconnect between raw social media data and standard natural language processing (NLP) tools.
When text is phonetically transliterated or mixed across scripts, it introduces immense orthographic variation, phonetic inconsistency, and character encoding mismatches. This fragmentation makes it incredibly difficult for standard models to recognize words or extract meaningful features.
The Mechanism of Script Normalization
To bridge this gap, researchers employ script normalization—a preprocessing pipeline that transforms diverse, informal, or mixed-script inputs into a consistent, unified representation. This normalization generally follows one of three strategies:
- Back-Transliteration to the Native Script: This involves mapping Romanized or phonetically transliterated text back into its original native script. For instance, Vishal Gupta et al. back-transliterate Roman-script Hindi into Devanagari, while Ruba Priyadharshini et al. convert Tamil words written in the English alphabet back into Tamil script.
- Homogenization into a Standardized Target Language: Instead of returning to the native script, some pipelines translate or transliterate the code-mixed utterances entirely into a single common language, such as English, to simplify downstream processing (as seen in the work of Ronghao Pan et al. and Mehrzad Tareh et al.).
- Unified Script or Phonetic Encodings: Other approaches map text into a shared intermediate representation. This includes the International Phonetic Alphabet (IPA) used by Judith Jeyafreeda Andrew, a standardized Romanized ASCII format used by Rada Mihalcea et al., or converting Traditional Chinese to Simplified Chinese to unify datasets as explored by Bin Wang et al. and Lanjun Zhou.
Why Script Normalization Works: Aligning Feature Spaces
Why is script normalization so effective compared to directly processing raw code-mixed text? The answer lies in how modern language models are pre-trained.
Large-scale pre-trained language models are exposed primarily to massive, high-quality, standardized monolingual corpora written in native scripts. They have far less exposure to the informal, phonetic transliterations common on social media. When a model encounters Romanized Hindi or Tamil, it suffers from severe vocabulary sparsity because the words appear unrecognizable.
By applying back-transliteration, the text is aligned with the exact script used during the model's pre-training phase. As Ayush Kumar et al. explain, converting Romanized text back to its native script allows models to better utilize pre-trained cross-lingual representations. This process reduces vocabulary sparsity and enables the model to recognize words written in foreign alphabets by leveraging standard monolingual embeddings, a mechanism highlighted by Kelsey Ball.
Furthermore, Vishal Gupta et al. demonstrate that back-transliteration preserves the original syntactic structure and semantic intent of code-mixed text far better than full machine translation, which often distorts or loses the nuanced syntax of the original query. By normalizing the script while maintaining the underlying language mix, models can process linguistically diverse inputs within a consistent feature space (Thamar Solorio et al.), unlocking the full potential of the model's pre-existing linguistic knowledge (Zaki Mustafa Farooqi).
Key Foundational Work
An Application Du Systeme A.T.E.F. A L’analyse Morphologique De Textes Russes (1973) — contributed 2 concept elements to this direction.