A Comparison of Pivot Methods for Phrase-Based Statistical Machine Translation

Masao Utiyama, Hitoshi IsaharaNAACL 2007

TL;DR

This paper explores two "pivot" strategies for machine translation when direct parallel data is scarce: phrase translation (building a new phrase table from two existing ones) and sentence translation (translating through an intermediate language sentence-by-sentence). They found that the phrase translation strategy significantly outperforms the sentence translation strategy and achieves competitive results compared to directly trained systems.

What Problem Does This Paper Address?

Building high-quality statistical machine translation (SMT) systems typically requires large parallel corpora – collections of text and its translation. However, such corpora are not available for all language pairs. This paper addresses the challenge of translating between languages when direct parallel data is scarce, but parallel data exists for the source language to a "pivot" language (like English) and for the pivot language to the target language. They propose and compare two methods to leverage these indirect parallel resources.

Methods

The authors compare two pivot strategies:

  • Phrase Translation Strategy: This method directly constructs a phrase translation table for the desired language pair (e.g., French-German) by combining two existing phrase tables: one for French-English and another for English-German. These phrase tables are built using standard phrase-based SMT techniques, including word alignments derived from tools like GIZA++. The resulting combined phrase table is then used within a standard phrase-based SMT system (Pharaoh).
  • Sentence Translation Strategy: This approach translates a source sentence into multiple English sentences, and then translates each of these English sentences into the target language. The system then selects the highest-scoring translation among the generated target sentences. This strategy relies on two independently trained SMT systems: one for the source-to-English translation and another for English-to-target translation.

The paper also details the underlying phrase-based SMT system used, which is based on a log-linear formulation and employs features like language model probabilities, phrase translation probabilities, and lexical translation probabilities. Minimum error rate training is used to tune the weights of these features.

Datasets

The experiments were conducted using the Europarl corpus. This corpus provided parallel data for six language pairs: Spanish-French, French-Spanish, German-French, French-German, German-Spanish, and Spanish-German. For training, they utilized data from the 2006 NAACL/HLT workshop on statistical machine translation, which included French-English, Spanish-English, and German-English parallel corpora. They extracted aligned sentences across all four languages using English as the pivot, resulting in approximately 585,830 sentences for each language pair.

Key Insights and Ideas

  • Phrase translation is more effective than sentence translation for pivot-based SMT: The core finding is that directly constructing a phrase translation table from intermediate tables is superior to translating sentence-by-sentence through a pivot language.
  • Phrase translation leverages the SMT decoder better: The phrase translation strategy allows the constructed phrase tables to be directly integrated into existing phrase-based SMT decoders, enabling them to exploit the decoder's full capabilities. This is a key advantage over the sentence translation strategy.
  • Recall is more important than precision for phrase table construction: The analysis of phrase table statistics suggests that the number of shared phrases (recall) is a more critical factor for translation performance than the exactness of those shared phrases (precision).
  • Direct training is still the gold standard: As expected, directly trained SMT systems (using direct parallel corpora) consistently outperformed all pivot-based methods. However, the phrase translation strategy achieved results that were very close to direct training.
  • Sentence translation's limitations: The sentence translation strategy's performance was hampered by the trade-off between the number of intermediate translations (n) and translation speed. A larger n could improve quality but drastically slows down the system, making it impractical.

Key Results

  • The Phrase Translation strategy significantly outperformed the Sentence Translation strategy across all tested language pairs.
  • The relative performance of the Phrase Translation strategy compared to directly trained SMT systems was 0.92 to 0.97 (meaning it achieved 92-97% of the BLEU score of the directly trained systems).
  • The Sentence Translation strategy (specifically SntTrans15) performed worse than PhraseTrans, with relative performances ranging from 0.82 to 0.96.
  • The Direct SMT systems achieved the highest BLEU scores, serving as an upper bound.
  • Table 1 shows that for Spanish-French, the Direct system scored 35.78 BLEU, PhraseTrans scored 32.90 (0.92 relative), SntTrans15 scored 29.49 (0.82 relative), and SntTrans1 scored 29.16 (0.81 relative). Similar trends were observed across other language pairs.

Limitations and Future Work

  • Controlled Environment: The experiments used the Europarl corpus, which is relatively large. The authors acknowledge that this is not a perfect simulation of the intended scenario where parallel data is extremely scarce. Future work will involve further investigation in truly low-resource situations.
  • Sentence Translation Speed: The sentence translation strategy suffers from significant speed limitations, making it impractical for large n.
  • Further Investigation: The authors plan to further investigate the pivot strategies to confirm their effectiveness in truly low-resource scenarios and explore ways to improve the sentence translation strategy, potentially by using larger n or more efficient methods. They also suggest that techniques for improving phrase translation table quality, such as translation disambiguation, could be beneficial.

Summary updated

BibTeX
@inproceedings{utiyama2007comparison,
  author = {Masao Utiyama and Hitoshi Isahara},
  title = {{A Comparison of Pivot Methods for Phrase-Based Statistical Machine Translation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2007},
}
Figure