Abstract
Out-of-vocabulary (OOV) words can pose serious challenges for machine translation (MT) tasks, and in particular, for low-resource language (LRL) pairs, i.e., language pairs for which few or no parallel corpora exist. Our work adapts variants of seq2seq models to perform transduction of such words from Hindi to Bhojpuri (an LRL instance), learning from a set of cognate pairs built from a bilingual dictionary of Hindi--Bhojpuri words. We demonstrate that our models can be effectively used for language pairs that have limited parallel corpora; our models work at the character level to grasp phonetic and orthographic similarities across multiple types of word adaptations, whether synchronic or diachronic, loan words or cognates. We describe the training aspects of several character level NMT systems that we adapted to this task and characterize their typical errors. Our method improves BLEU score by 6.3 on the Hindi-to-Bhojpuri translation task. Further, we show that such transductions can generalize well to other languages by applying it successfully to Hindi -- Bangla cognate pairs. Our work can be seen as an important step in the process of: (i) resolving the OOV words problem arising in MT tasks, (ii) creating effective parallel corpora for resource-constrained languages, and (iii) leveraging the enhanced semantic knowledge captured by word-level embeddings to perform character-level tasks.
Introduction
Standard Neural Machine Translation (NMT) systems are notoriously data-intensive, requiring massive parallel corpora to achieve high performance. For low-resource languages (LRLs) like Bhojpuri, such corpora are often unavailable, leading to a significant Out-of-Vocabulary (OOV) problem. When an NMT system encounters a word not present in its training data, translation quality degrades sharply.
However, many LRLs are closely related to high-resource "lingua franca" languages. For instance, Hindi and Bhojpuri share a vast number of cognates—words with common etymological origins and similar phonetic or orthographic structures. This paper proposes a "word transduction" strategy: instead of relying on sparse dictionaries, we can learn the systematic rules governing how words transform between related languages.
Methodology
The researchers framed word transduction as a character-level sequence-to-sequence task. They evaluated four primary NMT architectures: basic seq2seq, Alignment Models (AM) with Bahdanau-style attention, Hierarchical Attention Networks (HAN), and Transformer Networks (TN).
A key technical contribution is the use of novel pre-trained character embedding strategies. Two methods were explored: LM-LSTM-Embed, using a Bi-LSTM language model trained on Hindi Wikipedia, and fT-Avg-Embed, which derives character embeddings by averaging 300-dimensional Hindi fastText word vectors. These embeddings allow the model to leverage semantic and structural knowledge from the source language to improve character-level mapping.
Datasets and Evaluation
The models were trained and tested on several datasets:
- Hindi–Bhojpuri Cognates: 4,220 expert-curated word pairs.
- Hindi–Bangla Cognates: 3,220 pairs to test cross-language generalization.
- Bhojpuri MT Corpus: Approximately 40,000 Bhojpuri sentences scraped from blogs, paired with "noisy" Hindi translations created via English as a pivot.
Results and Discussion
The Transformer Network (TN) emerged as the superior architecture, achieving a 90.89 BLEU score and 75.71% Word Accuracy on the Hindi-Bhojpuri transduction task. Its success is attributed to residual connections that allow it to "copy" unchanged parts of a word while focusing attention on specific characters requiring modification (e.g., suffix changes).
In downstream translation tasks, integrating the transduction model as a fallback for OOV words led to a dramatic improvement. The baseline NMT system's score of 7.1 BLEU increased to 13.4 BLEU, representing a 6.3-point gain.
The study also found that "language distance" is a critical factor; performance was higher for the closely related Hindi-Bhojpuri pair than for the more distant Hindi-Bangla pair. Furthermore, pre-training character embeddings (specifically LM-LSTM-Embed) consistently outperformed random initialization, confirming that prior knowledge of source language structure is beneficial.
Conclusion
This work demonstrates that character-level transduction is a powerful tool for mitigating the OOV problem in low-resource MT. While the current approach is context-insensitive—transducing words in isolation without considering sentence-level grammar—it provides a robust foundation for future research. Subsequent efforts will focus on developing end-to-end MT pipelines and extending the method to a broader range of related language pairs.
Summary updated