Statistical Machine Translation in Low Resource Settings

Ann IrvineNAACL 2013

TL;DR

This paper tackles the challenge of improving statistical machine translation (SMT) for language pairs with limited parallel data. The authors propose integrating information from readily available monolingual and comparable corpora to reduce SMT's heavy reliance on expensive bitext, thereby expanding its applicability to more languages.

What Problem Does This Paper Address?

Modern Statistical Machine Translation (SMT) systems are notoriously data-hungry, requiring millions of lines of parallel text (bitext) to achieve good performance. This dependence creates a significant barrier for low-resource language pairs, where such extensive bitext is simply unavailable. When bitext is scarce, SMT models suffer from sparse word and phrase counts, leading to poor translation quality. This paper aims to bridge this gap by developing methods to make SMT viable for these low-resource scenarios, focusing on reducing the need for massive parallel datasets.

Methods

The paper proposes several key strategies to enhance SMT in low-resource settings:

  • Translating Unknown Words (OOVs):
    • Transliteration: Leveraging prior work, the authors plan to integrate transliteration for named entities and other words that are pronounced similarly across languages but have different spellings. They also aim to predict when transliteration is more appropriate than translation.
  • Bilingual Lexicon Induction: This involves learning word translation pairs from monolingual or comparable corpora. The authors propose a supervised approach, using a small amount of parallel data to extract an initial bilingual dictionary for positive supervision. They utilize various signals like orthographic similarity, topic similarity (from Wikipedia), temporal similarity (from news articles), and contextual similarity to build a discriminative classifier.
  • Inducing and Scoring Phrase Tables:
    • Phrase Translation Induction: To address the combinatorial explosion of phrases, the authors suggest using distributed phrase representations, employing filters to narrow down the search space (e.g., based on word translations, phrase lengths), and predicting when phrases should be treated as units rather than composed from individual word translations.
  • Phrase Translation Scoring: Similar to word induction, they propose using signals derived from comparable corpora (contextual, temporal, topical similarity for phrases, and lexical similarity for aligned words within phrases) to score phrase pairs and improve SMT models.

Datasets

The paper utilizes:

  • Monolingual Corpora: Web crawled time-stamped news articles and Wikipedia data for 22 languages. Wikipedia pages that are inter-lingually linked to English pages are specifically extracted.
  • Parallel Corpora: A small amount of parallel data is assumed to be available.
  • Existing Datasets: For preliminary results, the authors use Tamil, Bengali, and Hindi datasets released by Post et al. (2012), which were collected via Amazon's Mechanical Turk. These datasets are characterized by their small size and the high percentage of out-of-vocabulary (OOV) words in the development sets.

Key Insights and Ideas

  • Leveraging Monolingual Data is Crucial: The core idea is that abundant monolingual and comparable data can compensate for the lack of parallel data, a significant departure from traditional SMT approaches.
  • Supervised Lexicon Induction is Powerful: Using even a small parallel corpus for supervision dramatically improves bilingual lexicon induction compared to purely unsupervised methods.
  • Multiple Signals are Better: Combining various signals (orthographic, topic, temporal, contextual) for word and phrase translation induction leads to more robust and accurate results.
  • OOV Handling is Key: Effectively translating unknown words is a major bottleneck in low-resource SMT, and the proposed methods show promise in addressing this.
  • Phrase-Level Improvements: Beyond word translation, inducing and scoring phrase translations is identified as a critical step for further performance gains.

Key Results

Preliminary results show significant improvements:

  • Bilingual Lexicon Induction: The supervised approach achieved an average top-10 accuracy of 43.9%, a 44% improvement over the unsupervised baseline (30.4%).
  • OOV Translation Impact: Adding supervised word translations improved BLEU scores by approximately 0.24 to 0.79 points across Tamil, Bengali, and Hindi.
  • Combined Approach: Supplementing the model with both induced word translations and additional features from comparable corpora yielded additive BLEU improvements, with total gains of up to 1.25 for Tamil.
  • Top-K Translations: For Tamil and Hindi, using the top-5 ranked translations for OOV words provided a slight boost over using only the single best translation.

Limitations and Future Work

  • Transliteration for Non-Roman Scripts: While transliteration is discussed, adapting techniques for non-Roman scripts without off-the-shelf tools is noted as a challenge.
  • Phrase Induction Complexity: The combinatorial nature of phrase induction remains a significant challenge, and the paper proposes exploring ways to speed up this search.
  • Compositionality vs. Unit Translation: Further research is needed to accurately predict when phrases should be translated compositionally versus as a single unit.
  • Future Work: The thesis aims to extend these techniques to induce phrase translations and integrate them into phrase-based SMT models, ultimately broadening SMT's applicability to more language pairs and domains.

Summary updated

BibTeX
@inproceedings{irvine2013statistical,
  author = {Ann Irvine},
  title = {{Statistical Machine Translation in Low Resource Settings}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2013},
}
Figure