TL;DR
Researchers have developed a novel data selection method called Scaled Similarity Score (SSS) and a transliteration trick to boost machine translation for low-resource languages like Nepali. By leveraging the linguistic closeness of Hindi and Nepali and filtering out-of-domain data more effectively, they achieved improvements of up to 3 BLEU points over standard techniques.
What Problem Does This Paper Address?
Machine Translation (MT) is a data-hungry field. While systems for English-German are world-class, "low-resource" languages (LRLs)—those without massive parallel corpora—struggle to reach usable quality.
A common strategy is Domain Adaptation (DA): taking a model trained on a general or "out-of-domain" dataset (like the Bible or technical manuals) and tweaking it for a specific "in-domain" task (like news or daily conversation). However, standard data selection methods often fail when data is extremely scarce. This paper tackles the specific challenge of translating between Hindi and Nepali, two closely related languages that share the Devanagari script, similar word orders, and many cognates, but lack the massive datasets required for traditional Neural Machine Translation (NMT).
Methods
The authors propose three key innovations to bridge the resource gap:
- WX-Transliteration for Language Modeling: Before training their filtering models, they convert the Devanagari script into WX-notation (a transliteration scheme for Indian languages). This reduces the "orthographic distance" between Hindi and Nepali, making it easier for a language model to recognize that the two languages are linguistically similar.
- Scaled Similarity Score (SSS): Traditional data selection often uses cross-entropy to find sentences in a large "out-of-domain" pool that look like "in-domain" data. The authors instead use SSS, which scales the language model scores between 0 and 1. This normalization helps the model favor sentences that are strictly closer to the target domain, filtering out noise more effectively.
- Integrated DA Techniques: They applied SSS across three common training workflows:
- Multi-Domain Approach (MDA): Training on a mix of domains.
- Fine-Tuning Approach (FTA): Training on out-of-domain data first, then specialized in-domain data.
- Iterative Back-Translation (SSSIBA): Using the model to translate monolingual text to create synthetic parallel data, but using SSS to filter only the best synthetic pairs.
Datasets
The study utilized several datasets across different domains:
- In-domain: Hindi-Nepali parallel data from the WMT 2019 shared task (~65k sentences).
- Out-of-domain: Agriculture (AGRI), Entertainment (ENT), and Bible (BIB) corpora sourced from TDIL and OPUS.
- Monolingual: 90k sentences each for Hindi and Nepali to support back-translation.
Key Insights and Ideas
- Transliteration is a "Cheat Code": By using WX-notation, the perplexity of their KenLM language model dropped drastically (e.g., from 477.3 down to 8.1 for Hindi). This suggests that for related languages, the script often masks how similar the underlying languages actually are.
- Scaling Matters: Raw cross-entropy scores can have massive gaps between minimum and maximum values. Scaling these into the SSS (0 to 1 range) prevents outliers from dominating the selection process and ensures the model picks sentences that truly represent the target domain.
- SMT Still Has Legs: Interestingly, the paper notes that Statistical Machine Translation (SMT) can sometimes outperform NMT on very limited datasets for these language pairs, likely because SMT handles orthographic similarity and word-for-word overlaps more robustly when data is too thin for deep learning.
Key Results
The proposed SSS method consistently outperformed the state-of-the-art data selection methods (like Moore-Lewis filtering):
- Multi-domain: Gained ~2 BLEU points.
- Fine-tuning: Gained ~3 BLEU points.
- Iterative Back-Translation: Gained ~2 BLEU points.
- Qualitative Similarity: The authors demonstrated that domains with higher "overlapping vocabulary" (like Entertainment) naturally performed better than those with low overlap (like the Bible), confirming that lexical similarity is a primary driver of MT quality in this setup.
Limitations and Future Work
The primary limitation is that this approach relies on language relatedness. It works beautifully for Hindi-Nepali because they share a script and origin; it might be less effective for unrelated language pairs (e.g., Hindi to English).
The authors suggest that future work should explore dynamic data selection—adjusting the filtering threshold during the training process—and expanding the technique to cross-lingual transfer learning involving multiple similar languages simultaneously (e.g., using Hindi and Marathi to improve Nepali translation).
Summary updated