Predicting Success in Machine Translation

Alexandra Birch, Miles Osborne, Philipp KoehnEMNLP 2008 · OpenReview

Abstract

The performance of machine translation (MT) systems varies significantly across different language pairs. This study investigates the impact of linguistic characteristics on the performance of phrase-based SMT for 110 European language pairs. By analyzing word reordering, morphological complexity, and historical relatedness, we demonstrate that these three factors explain 75% of the variation in translation quality.

Introduction: The Performance Gap

In the development of Statistical Machine Translation (SMT), researchers observed a persistent performance gap between language pairs. While systems translating between English and French achieved high accuracy, those involving languages like Finnish or German lagged significantly. While linguistic intuition suggested that grammar and word order were to blame, these claims remained largely anecdotal. This research aims to quantify which specific linguistic variables predict MT success.

Methodology

The study utilizes multiple linear regression to model the relationship between linguistic "explanatory variables" and BLEU scores across 110 language pairs from the Europarl corpus. Three primary features were analyzed:

  1. Morphological Complexity: Measured by vocabulary size.
  2. Language Relatedness: Measured via lexicostatistics (cognate counts).
  3. Reordering (RQuantity): A metric quantifying the structural shifts between source and target sentences.

Key Findings

The Impact of Morphological Complexity

A critical insight from the study is that the morphological complexity of the target language is a far stronger predictor of MT failure than that of the source language. It is significantly harder for a model to generate correct inflections in a morphologically rich language than it is to parse them.

Average vocabulary size across 11 languages, highlighting Finnish as a morphological outlier.

As shown in Figure 1, Finnish serves as a primary example of this challenge. Its massive vocabulary size, resulting from its agglutinative nature, makes it the most difficult target language in the dataset.

The Challenge of Word Reordering

Reordering (RQuantity) measures how much the word order must change during translation. The study found that reordering is a primary bottleneck for translation quality. When extreme outliers like Finnish are removed, the amount of reordering required becomes the single most significant predictor of translation difficulty.

Matrix of translation performance (BLEU scores) across 110 source-target language pairs.

Figure 5 visualizes the performance across the 110 pairs. Closely related languages, such as the Romance group (Italian, Spanish, Portuguese, French), show higher performance (larger squares) due to shared structure and high cognate counts.

The 75% Rule and Relatedness

By combining reordering, target complexity, and historical relatedness, the regression model achieved an $R^2$ of 0.75. This indicates that three-quarters of the performance variance in MT is predictable based on these linguistic hurdles. Source vocabulary size, by contrast, showed a negligible correlation ($R^2 = 0.045$), suggesting that sparse counts in the source text are a minor issue compared to target-side generation.

Conclusion

The "difficulty" of machine translation is not an enigma but a direct function of quantifiable linguistic differences. While source language complexity has a negligible impact, the target language's morphology and the structural divergence between the pair are decisive. These findings provide a roadmap for where MT improvements—such as better inflection generation or more robust reordering models—are most needed.

Note: This study was conducted using phrase-based SMT. While these linguistic factors remain relevant, the advent of Neural Machine Translation (NMT) may interact with these variables differently.

Summary updated

Abstract

The performance of machine translation systems varies greatly depending on the source and target languages involved. Determining the contribution of different characteristics of language pairs on system performance is key to knowing what aspects of machine translation to improve and which are irrelevant. This paper investigates the effect of different explanatory variables on the performance of a phrase-based system for 110 European language pairs. We show that three factors are strong predictors of performance in isolation: the amount of reordering, the morphological complexity of the target language and the historical relatedness of the two languages. Together, these factors contribute 75% to the variability of the performance of the system.
BibTeX
@inproceedings{birch2008predicting,
  author = {Alexandra Birch and Miles Osborne and Philipp Koehn},
  title = {{Predicting Success in Machine Translation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2008},
  url = {https://openreview.net/forum?id=BJbSJGGuWH},
}
Figure