Abstract
The performance of machine translation (MT) systems varies significantly across different language pairs. This study investigates the impact of linguistic characteristics on the performance of phrase-based SMT for 110 European language pairs. By analyzing word reordering, morphological complexity, and historical relatedness, we demonstrate that these three factors explain 75% of the variation in translation quality.
Introduction: The Performance Gap
In the development of Statistical Machine Translation (SMT), researchers observed a persistent performance gap between language pairs. While systems translating between English and French achieved high accuracy, those involving languages like Finnish or German lagged significantly. While linguistic intuition suggested that grammar and word order were to blame, these claims remained largely anecdotal. This research aims to quantify which specific linguistic variables predict MT success.
Methodology
The study utilizes multiple linear regression to model the relationship between linguistic "explanatory variables" and BLEU scores across 110 language pairs from the Europarl corpus. Three primary features were analyzed:
- Morphological Complexity: Measured by vocabulary size.
- Language Relatedness: Measured via lexicostatistics (cognate counts).
- Reordering (RQuantity): A metric quantifying the structural shifts between source and target sentences.
Key Findings
The Impact of Morphological Complexity
A critical insight from the study is that the morphological complexity of the target language is a far stronger predictor of MT failure than that of the source language. It is significantly harder for a model to generate correct inflections in a morphologically rich language than it is to parse them.
As shown in Figure 1, Finnish serves as a primary example of this challenge. Its massive vocabulary size, resulting from its agglutinative nature, makes it the most difficult target language in the dataset.
The Challenge of Word Reordering
Reordering (RQuantity) measures how much the word order must change during translation. The study found that reordering is a primary bottleneck for translation quality. When extreme outliers like Finnish are removed, the amount of reordering required becomes the single most significant predictor of translation difficulty.
Figure 5 visualizes the performance across the 110 pairs. Closely related languages, such as the Romance group (Italian, Spanish, Portuguese, French), show higher performance (larger squares) due to shared structure and high cognate counts.
The 75% Rule and Relatedness
By combining reordering, target complexity, and historical relatedness, the regression model achieved an $R^2$ of 0.75. This indicates that three-quarters of the performance variance in MT is predictable based on these linguistic hurdles. Source vocabulary size, by contrast, showed a negligible correlation ($R^2 = 0.045$), suggesting that sparse counts in the source text are a minor issue compared to target-side generation.
Conclusion
The "difficulty" of machine translation is not an enigma but a direct function of quantifiable linguistic differences. While source language complexity has a negligible impact, the target language's morphology and the structural divergence between the pair are decisive. These findings provide a roadmap for where MT improvements—such as better inflection generation or more robust reordering models—are most needed.
Note: This study was conducted using phrase-based SMT. While these linguistic factors remain relevant, the advent of Neural Machine Translation (NMT) may interact with these variables differently.
Summary updated