Introduction
Code-switching—the practice of mixing multiple languages in a single conversation—is ubiquitous among multilingual speakers. On social media, Hindi speakers frequently use the Roman (Latin) script to write Hindi words due to the relative ease of use compared to Devanagari keyboards. However, this "romanization" is highly non-standard, characterized by ad-hoc phonetic rules, vowel dropping (e.g., yaar as yr), and ambiguous consonant usage.
This creates a significant hurdle for Natural Language Processing (NLP) systems. State-of-the-art models like mBERT, typically trained on formal Devanagari text, often fail to recognize Hindi components in romanized strings like "bhai...why r u crying," leading to degraded performance in downstream tasks such as Named Entity Recognition (NER) and Part-of-Speech (POS) tagging.
Methodology
The authors propose a robust pipeline to bridge the gap between noisy romanized input and standardized native-script representations.
Normalization
The first and most critical step treats normalization as a machine translation problem. Using a character-level Sequence-to-Sequence (Seq2Seq) model with an LSTM encoder-decoder architecture, the system transforms informal romanized tokens into a standardized romanized form. This step resolves common social media abbreviations and phonetic inconsistencies.
Back-Transliteration and G2P
Once normalized, the text is syllabified and converted into Devanagari script using a mapping table. By normalizing first, the system can rely on simpler phonetic mappings rather than complex deep learning for the script conversion itself. Additionally, the authors developed a Grapheme-to-Phoneme (G2P) technique to map normalized tokens to the International Phonetic Alphabet (IPA), extending the Epitran library with custom Hindi mappings.
Datasets and Evaluation
The normalization model was trained on the hinglishNorm dataset, augmented with common social media variations. Back-transliteration was evaluated against the Xlit-Crowd corpus. Furthermore, the authors contributed script-corrected versions of the LinCE benchmark (NER and POS tagging subsets) to the research community.
Key Results
The proposed system demonstrates significant improvements over existing baselines. The Seq2Seq normalization model achieved a BLEU score of 80.48 and a METEOR score of 0.56, outperforming prior work in the field.
In the back-transliteration task, the system reached 78.6% accuracy on the Xlit-Crowd dataset, notably surpassing the popular indic-trans library, which scored 63.56%.
Discussion and Conclusion
A central insight of this work is that normalization is the primary bottleneck in processing code-switched data. Once informal spelling variations are mitigated, transliteration becomes straightforward. Furthermore, the authors highlight that script matters: models like mBERT show a substantial increase in zero-shot performance when Hindi words are presented in Devanagari rather than Roman script. This suggests that the visual representation of a language is as crucial as its semantic understanding for multilingual models. Future work will focus on dictionary augmentation and extending this methodology to other Devanagari-based languages like Gujarati and Nepali.
Summary updated