Learning cross-lingual phonological and orthagraphic adaptations: a case study in improving neural machine translation between low-resource languages

Saurav Jha, Akhilesh Sudhakar, Anil Kumar Singh

TL;DR

This paper tackles the challenge of Out-of-Vocabulary (OOV) words in low-resource language machine translation (MT) by introducing a "word transduction" approach. By training character-level Neural Machine Translation (NMT) models on cognate pairs between Hindi and Bhojpuri, they significantly improve MT performance, demonstrating a practical solution for languages with limited parallel data.

What Problem Does This Paper Address?

Machine translation (MT) systems, especially data-hungry Neural Machine Translation (NMT) models, struggle with Out-of-Vocabulary (OOV) words – words not present in their training vocabulary. This problem is particularly acute for Low-Resource Languages (LRLs), where parallel corpora (pairs of sentences in two languages) are scarce, leading to a high proportion of OOV words. Existing NMT models often fail to translate these words accurately, degrading overall translation quality. The authors aim to address this by developing a method to "transduce" OOV words, essentially predicting their target language equivalent based on phonetic and orthographic similarities, rather than relying on traditional translation.

Methods

The core of the paper is adapting character-level NMT models for word transduction. Instead of translating whole sentences, the models are trained to transform individual Hindi words into their Bhojpuri counterparts. This is achieved by:

  • Character-level NMT: Utilizing sequence-to-sequence (seq2seq) models that operate at the character level. This allows the models to capture fine-grained phonetic and orthographic similarities between words, which is crucial for identifying cognates and predicting word adaptations.
  • Cognate Pair Training: The models are trained exclusively on pairs of words that are cognates (share a common origin and meaning) between Hindi and Bhojpuri. This is a key distinction from standard MT, which uses parallel sentences.
  • Model Architectures: The authors experiment with several NMT architectures:
  • A simple seq2seq model.
  • An Alignment Model (AM) with attention.
  • A Hierarchical Attention Network (HAN).
  • A Transformer Network (TN).
  • Character Embeddings: They explore two pre-training strategies for character embeddings:
  • LM-LSTM-Embed: Training a character-level language model on Hindi Wikipedia articles.
  • fT-Avg-Embed: Deriving character embeddings by averaging pre-trained Hindi word embeddings from fastText.

Datasets

  • Hindi-Bhojpuri Cognate Pairs: The primary dataset consists of 4220 Hindi-Bhojpuri word cognate pairs, compiled by linguistic experts. This dataset was split into training (67.5%), validation (7.5%), and test (25%) sets.
  • Hindi-Bhojpuri Parallel Corpus (for MT experiments): An artificial parallel corpus of approximately 40,000 Hindi-Bhojpuri sentence pairs was created using Google Translate API via English as an intermediate language. This was used to evaluate the impact of their transduction method on overall MT performance.
  • Hindi-Bangla Cognate Pairs: An additional dataset of 3220 Hindi-Bangla cognate pairs was used to test the generalizability of their approach.

Key Insights and Ideas

  • Cognates as a Bridge: The fundamental idea is that for closely related languages like Hindi and Bhojpuri, cognates (words with shared origins and similar forms) are abundant and can serve as a powerful signal for word adaptation, even when direct translation data is scarce.
  • Character-level is Key for Adaptation: Operating at the character level allows models to learn subtle phonetic and orthographic transformations that are characteristic of word adaptations (e.g., sound shifts, spelling variations) between related languages.
  • Transduction vs. Translation: The paper clearly distinguishes "transduction" (learning from cognates) from "translation" (learning from parallel sentences). Transduction is presented as a more focused task that can be more effective for OOV words in LRLs.
  • Generalizability: The success of the Hindi-Bhojpuri transduction models on Hindi-Bangla pairs suggests that the learned character-level adaptation mechanisms can generalize to other language pairs with sufficient cognate overlap.
  • Transformer's Strength: The Transformer model (TN) emerged as the top performer, attributed to its residual connections and multi-head attention mechanism, which effectively capture long-range dependencies between characters.

Key Results

  • Improved BLEU Score: The proposed word transduction method improved the BLEU score by 6.3 points on the Hindi-to-Bhojpuri translation task when applied to OOV words. The overall system achieved a BLEU score of 13.4.
  • State-of-the-Art Performance: The Transformer model (TN) achieved the best performance across all metrics, outperforming the previous state-of-the-art (Sharma and Singh, 2017) by 11.07 BLEU score points and 11.3% word accuracy.
  • Character Embeddings Matter: LM-LSTM-Embed pre-trained character embeddings consistently yielded better transduction performance than fT-Avg-Embed or random initialization.
  • Model Performance: The Transformer (TN) generally outperformed AM and HAN, with the simple seq2seq model performing the worst. HAN showed a tendency to produce unpronounceable translations.

Limitations and Future Work

  • Language Distance: The effectiveness of the cognate-based transduction method is inherently limited by the "language distance." While it works well for closely related languages like Hindi-Bhojpuri, the number of cognates decreases significantly for more distant pairs like Hindi-Bangla, impacting performance.
  • Data Scarcity for Distant Pairs: The limited number of cognates for more distant language pairs poses a challenge for training robust transduction models.
  • Future Work:
    • Investigate the method on more closely related language pairs.
  • Build state-of-the-art MT pipelines for LRLs incorporating their OOV handling method.
  • Compare transduction with character-level MT systems for OOV problems.
  • Explore a primitive yet useful MT method using Google Translate APIs for languages with no known MT systems.

Summary updated

Abstract

Out-of-vocabulary (OOV) words can pose serious challenges for machine translation (MT) tasks, and in particular, for low-resource language (LRL) pairs, i.e., language pairs for which few or no parallel corpora exist. Our work adapts variants of seq2seq models to perform transduction of such words from Hindi to Bhojpuri (an LRL instance), learning from a set of cognate pairs built from a bilingual dictionary of Hindi - Bhojpuri words. We demonstrate that our models can be effectively used for language pairs that have limited parallel corpora; our models work at the character level to grasp phonetic and orthographic similarities across multiple types of word adaptations, whether synchronic or diachronic, loan words or cognates. We describe the training aspects of several character level NMT systems that we adapted to this task and characterize their typical errors. Our method improves BLEU score by 6.3 on the Hindi-to-Bhojpuri translation task. Further, we show that such transductions can generalize well to other languages by applying it successfully to Hindi - Bangla cognate pairs. Our work can be seen as an important step in the process of: (i) resolving the OOV words problem arising in MT tasks; (ii) creating effective parallel corpora for resource constrained languages; and (iii) leveraging the enhanced semantic knowledge captured by word-level embeddings to perform character-level tasks.
BibTeX
@article{jha2019learning,
  author = {Saurav Jha and Akhilesh Sudhakar and Anil Kumar Singh},
  title = {{Learning cross-lingual phonological and orthagraphic adaptations: a case study in improving neural machine translation between low-resource languages}},
  journal = {J. Lang. Model.},
  year = {2019},
  url = {https://openreview.net/forum?id=mN4p9Wq1iB},
}
Figure