On the Importance of Pivot Language Selection for Statistical Machine Translation

Michael Paul, Hirofumi Yamamoto, Eiichiro Sumita, Satoshi NakamuraNAACL 2009

TL;DR

This paper investigates using non-English languages as "pivot" languages for machine translation, aiming to improve translation quality for language pairs with limited parallel data. Surprisingly, they found that for over half of the tested language pairs, a non-English pivot language actually yielded better results than the commonly used English pivot.

What Problem Does This Paper Address?

Statistical Machine Translation (SMT) systems thrive on large amounts of parallel text data (sentences and their translations). For widely spoken languages like English, French, or German, such data is abundant. However, for many "under-resourced" language pairs, especially those involving Asian languages, this data is scarce, leading to poor translation quality. To bridge this gap, researchers have turned to "pivot translation," where translation occurs indirectly through an intermediate language. The common practice is to use English as this pivot due to its vast linguistic resources. This paper questions this assumption and explores whether other languages might be more effective pivots, particularly for improving translations involving under-resourced languages.

Methods

The core of this research involves conducting extensive pivot translation experiments. They employed a standard phrase-based SMT framework. The pivot translation itself was implemented using a "cascading" approach: first, a source-to-pivot (SRC-PVT) translation system was trained, and then a pivot-to-target (PVT-TRG) system was trained. The output of the first system became the input for the second.

To isolate the effect of the pivot language, they used a specific dataset (BTEC) and randomly split it into two 80k sentence subsets for training SRC-PVT and PVT-TRG models. This ensured they weren't relying on a single, large, trilingual corpus, which might not be realistic for under-resourced pairs. They evaluated translation quality using standard metrics like BLEU and METEOR.

Datasets

The primary dataset used was the Basic Travel Expressions Corpus (BTEC). This corpus consists of sentences considered useful for travelers. The authors selected twelve major world languages from BTEC, including Indo-European and Asian languages, to cover a diverse range of linguistic characteristics (word order, inflection, etc.). The BTEC corpus was split into training (80k sentence pairs for SRC-PVT and 80k for PVT-TRG), development (dev), and evaluation (eval) sets. They also conducted experiments with smaller training sets (10k sentences) to investigate data size dependency.

Key Insights and Ideas

  • English isn't always best: The most striking finding is that English is not the optimal pivot language for the majority of language pairs tested. This challenges the conventional wisdom in the field.
  • Asian languages as pivots: Non-English pivot languages, particularly Asian languages like Japanese, Malay, Indonesian, and Korean, proved superior for a significant portion of the language pairs.
  • Pivot choice is asymmetric: The optimal pivot language for translating from language A to language B might be different from the optimal pivot for translating from B to A. This highlights the nuanced relationship between language pairs and the pivot.
  • Relatedness matters, but how? The choice of pivot language is influenced by the translation quality of both the SRC-PVT and PVT-TRG systems. For smaller training sets, proximity to the source language seems more critical, while for larger sets, proximity to the target language becomes more important. This suggests a complex interplay between language characteristics and system performance.
  • Error chaining is a concern: Pivot translation can suffer from "error chaining," where errors in the first translation step propagate and degrade the final output. However, in some cases, the benefits of a well-chosen pivot outweigh this drawback.

Key Results

  • For 61 out of 110 language pairs tested, a non-English pivot language resulted in improved translation quality compared to using English as the pivot.
  • When English was the pivot, it performed best for languages that have high direct translation quality with English (e.g., German, Spanish, French).
  • For language pairs involving Japanese, Korean, Indonesian, or Malay, English was the optimal pivot in only about 16.2% of cases.
  • The optimal pivot language selection was consistent between small (10k) and large (80k) training data sets for about 75.5% of the translation tasks.

Limitations and Future Work

  • Limited dataset: The experiments were conducted on the BTEC corpus, which is specific to travel-related expressions. The findings might not generalize to all types of text.
  • Single pivot method: Only the cascading pivot translation method was investigated. Other methods like phrase-table composition or pseudo corpus generation might yield different results.
  • Feature engineering for pivot selection: The authors plan to investigate what specific linguistic features (e.g., word order, morphological complexity, historical relatedness) are strong predictors for optimal pivot language selection.
  • Automatic pivot prediction: The ultimate goal is to develop statistical classification models that can automatically predict the best pivot language for any given language pair, thereby improving the usability of machine translation for under-resourced languages.

Summary updated

BibTeX
@inproceedings{paul2009importance,
  author = {Michael Paul and Hirofumi Yamamoto and Eiichiro Sumita and Satoshi Nakamura},
  title = {{On the Importance of Pivot Language Selection for Statistical Machine Translation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2009},
}
Figure