TL;DR
This paper introduces a method to boost Neural Machine Translation (NMT) for low-resource Indian languages by projecting diverse scripts into a shared phonetic-orthographic space using WX notation. By converting different scripts (like Devanagari, Gurumukhi, and Perso-Arabic) into a common Latin-based encoding before subword tokenization, the authors enable NMT models to better exploit the inherent linguistic similarities between related languages, resulting in translation quality improvements of up to 10 BLEU points.
What Problem Does This Paper Address?
Translating between Indian languages is a "low-resource" challenge. While these languages are spoken by millions, they often lack the massive parallel corpora (paired sentences) required to train high-quality NMT models.
Furthermore, Indian languages are morphologically rich—a single root word can take many forms—and use a variety of scripts. Even though languages like Hindi, Punjabi, and Urdu share deep phonetic and lexical roots, their different writing systems prevent standard NMT models from "seeing" these similarities. A model trained on Devanagari (Hindi) won't naturally recognize the same word written in Perso-Arabic (Urdu), leading to fragmented vocabularies and poor performance in data-scarce scenarios.
Methods
The authors propose a three-step pipeline to create a common "character space" for translation:
- WX Encoding (The Core Innovation): Instead of feeding raw scripts into the model, they transliterate both source and target text into WX notation. WX is an ASCII-based transliteration scheme specifically designed for Indian languages. Crucially, it maps phonetically identical sounds to the same Latin characters across different scripts.
- Subword Tokenization: They apply Byte Pair Encoding (BPE) via the SentencePiece library on the WX-encoded text. Because the languages now share a common alphabet, BPE can identify shared subwords (roots, prefixes, suffixes) across the language pair, effectively "compressing" the morphological complexity.
- Transformer-based NMT: The processed text is used to train a standard Transformer model. After translation, the WX-encoded output is decoded back into the original target script.
The authors also explored Back-translation (generating synthetic parallel data from monolingual text) as an additional way to bolster performance in extremely low-resource settings.
Datasets
The study utilized several prominent repositories for Indian language data:
- CVIT-PIB: Parallel corpora from the Press Information Bureau.
- WMT Shared Tasks (2019/2020): Standard benchmarks for machine translation.
- OPUS & TDIL: Large-scale open-source and government-maintained language repositories.
- Language Pairs: The primary focus was on Hindi paired with Gujarati, Marathi, Nepali, Maithili, Punjabi, and Urdu.
- Distant/Zero-Shot: They also tested the approach on Dravidian languages (Malayalam, Tamil, Telugu) and zero-shot pairs (Bhojpuri and Magahi) where no parallel training data was available.
Key Insights and Ideas
- Phonetics as a Bridge: Most Indian scripts are Brahmi-derived and highly phonetic. By focusing on how words sound (via WX) rather than how they look (via Unicode), the model can leverage "cross-lingual transfer."
- The Urdu-Hindi Case: Urdu and Hindi are linguistically almost identical in spoken form but use entirely different scripts. WX notation effectively "unifies" them, allowing the model to treat them as the same language with different surface forms.
- Reducing Surface Complexity: The authors used entropy analysis to show that WX notation, combined with BPE, reduces the "redundancy" of the text. This makes it easier for the neural network to learn the underlying mapping between languages without getting bogged down by script-specific quirks.
Key Results
The proposed WX-based approach consistently outperformed baseline NMT models:
- Significant Gains: The most dramatic improvement was seen in Nepali $\leftrightarrow$ Hindi translation, with a jump of ~10 BLEU points.
- Consistent Wins: Most other pairs (Gujarati, Marathi, Punjabi, Urdu) saw improvements ranging from +1 to +5 BLEU points.
- Zero-Shot Success: The model showed a +1 BLEU point improvement on zero-shot languages like Bhojpuri, proving that the shared phonetic space helps the model generalize to languages it hasn't explicitly seen in training.
- Back-translation Synergy: Combining WX encoding with back-translation provided an additional boost, particularly for Gujarati $\leftrightarrow$ Hindi.
Limitations and Future Work
- Script Dependency: The method relies on the availability of high-quality WX converters. If a language lacks a robust transliteration mapping, this approach cannot be easily applied.
- Distant Language Families: While the method helped with Indo-Aryan to Dravidian translation (e.g., Hindi to Tamil), the improvements were less pronounced than those between closely related Indo-Aryan languages.
- Future Directions: The authors plan to extend this to a Many-to-Many Multilingual NMT system, where all Indian languages are projected into a single WX-based space. They also aim to incorporate more advanced pre-training techniques like mBART specifically adapted for this shared phonetic-orthographic space.
Summary updated