TL;DR
This paper tackles the challenge of Out-of-Vocabulary (OOV) words in low-resource language machine translation (MT) by introducing a "word transduction" approach. By training character-level Neural Machine Translation (NMT) models on cognate pairs between Hindi and Bhojpuri, they significantly improve MT performance, demonstrating a practical solution for languages with limited parallel data.
What Problem Does This Paper Address?
Machine translation (MT) systems, especially data-hungry Neural Machine Translation (NMT) models, struggle with Out-of-Vocabulary (OOV) words – words not present in their training vocabulary. This problem is particularly acute for Low-Resource Languages (LRLs), where parallel corpora (pairs of sentences in two languages) are scarce, leading to a high proportion of OOV words. Existing NMT models often fail to translate these words accurately, degrading overall translation quality. The authors aim to address this by developing a method to "transduce" OOV words, essentially predicting their target language equivalent based on phonetic and orthographic similarities, rather than relying on traditional translation.
Methods
The core of the paper is adapting character-level NMT models for word transduction. Instead of translating whole sentences, the models are trained to transform individual Hindi words into their Bhojpuri counterparts. This is achieved by:
- Character-level NMT: Utilizing sequence-to-sequence (seq2seq) models that operate at the character level. This allows the models to capture fine-grained phonetic and orthographic similarities between words, which is crucial for identifying cognates and predicting word adaptations.
- Cognate Pair Training: The models are trained exclusively on pairs of words that are cognates (share a common origin and meaning) between Hindi and Bhojpuri. This is a key distinction from standard MT, which uses parallel sentences.
- Model Architectures: The authors experiment with several NMT architectures:
- A simple seq2seq model.
- An Alignment Model (AM) with attention.
- A Hierarchical Attention Network (HAN).
- A Transformer Network (TN).
- Character Embeddings: They explore two pre-training strategies for character embeddings:
- LM-LSTM-Embed: Training a character-level language model on Hindi Wikipedia articles.
- fT-Avg-Embed: Deriving character embeddings by averaging pre-trained Hindi word embeddings from fastText.
Datasets
- Hindi-Bhojpuri Cognate Pairs: The primary dataset consists of 4220 Hindi-Bhojpuri word cognate pairs, compiled by linguistic experts. This dataset was split into training (67.5%), validation (7.5%), and test (25%) sets.
- Hindi-Bhojpuri Parallel Corpus (for MT experiments): An artificial parallel corpus of approximately 40,000 Hindi-Bhojpuri sentence pairs was created using Google Translate API via English as an intermediate language. This was used to evaluate the impact of their transduction method on overall MT performance.
- Hindi-Bangla Cognate Pairs: An additional dataset of 3220 Hindi-Bangla cognate pairs was used to test the generalizability of their approach.
Key Insights and Ideas
- Cognates as a Bridge: The fundamental idea is that for closely related languages like Hindi and Bhojpuri, cognates (words with shared origins and similar forms) are abundant and can serve as a powerful signal for word adaptation, even when direct translation data is scarce.
- Character-level is Key for Adaptation: Operating at the character level allows models to learn subtle phonetic and orthographic transformations that are characteristic of word adaptations (e.g., sound shifts, spelling variations) between related languages.
- Transduction vs. Translation: The paper clearly distinguishes "transduction" (learning from cognates) from "translation" (learning from parallel sentences). Transduction is presented as a more focused task that can be more effective for OOV words in LRLs.
- Generalizability: The success of the Hindi-Bhojpuri transduction models on Hindi-Bangla pairs suggests that the learned character-level adaptation mechanisms can generalize to other language pairs with sufficient cognate overlap.
- Transformer's Strength: The Transformer model (TN) emerged as the top performer, attributed to its residual connections and multi-head attention mechanism, which effectively capture long-range dependencies between characters.
Key Results
- Improved BLEU Score: The proposed word transduction method improved the BLEU score by 6.3 points on the Hindi-to-Bhojpuri translation task when applied to OOV words. The overall system achieved a BLEU score of 13.4.
- State-of-the-Art Performance: The Transformer model (TN) achieved the best performance across all metrics, outperforming the previous state-of-the-art (Sharma and Singh, 2017) by 11.07 BLEU score points and 11.3% word accuracy.
- Character Embeddings Matter: LM-LSTM-Embed pre-trained character embeddings consistently yielded better transduction performance than fT-Avg-Embed or random initialization.
- Model Performance: The Transformer (TN) generally outperformed AM and HAN, with the simple seq2seq model performing the worst. HAN showed a tendency to produce unpronounceable translations.
Limitations and Future Work
- Language Distance: The effectiveness of the cognate-based transduction method is inherently limited by the "language distance." While it works well for closely related languages like Hindi-Bhojpuri, the number of cognates decreases significantly for more distant pairs like Hindi-Bangla, impacting performance.
- Data Scarcity for Distant Pairs: The limited number of cognates for more distant language pairs poses a challenge for training robust transduction models.
- Future Work:
- Investigate the method on more closely related language pairs.
- Build state-of-the-art MT pipelines for LRLs incorporating their OOV handling method.
- Compare transduction with character-level MT systems for OOV problems.
- Explore a primitive yet useful MT method using Google Translate APIs for languages with no known MT systems.
Summary updated