The WMT’18 Morpheval test suites for English-Czech, English-German, English-Finnish and Turkish-English

Franck Burlot, Yves Scherrer, Vinit Ravishankar, Ondřej Bojar, Stig-Arne Grönroos, Maarit Koponen, Tommi Nieminen, François YvonEMNLP 2018

TL;DR

This paper introduces and applies new test suites for evaluating the morphological accuracy of machine translation systems across four language pairs. By extending existing "Morpheval" tests, the authors provide a more granular look at MT performance beyond traditional metrics, revealing interesting discrepancies between system rankings and their ability to handle linguistic nuances.

What Problem Does This Paper Address?

Neural Machine Translation (NMT) has achieved remarkable fluency, but its "black box" nature makes it hard to pinpoint why translations succeed or fail. Traditional metrics like BLEU offer a single score but don't reveal specific weaknesses. This is particularly problematic for morphologically rich languages where subtle changes in word endings, agreement, or tense can significantly alter meaning. The authors aim to address this by developing and applying fine-grained evaluation metrics that specifically target morphological competence, going beyond overall translation quality to assess how well MT systems handle linguistic variations.

Methods

The core of this work is the extension and application of the "Morpheval" protocol, originally developed for English-Czech and English-Latvian. The Morpheval approach involves creating "contrast pairs" of sentences where the source sentences differ by a single morphological feature (e.g., singular vs. plural noun, present vs. past tense). The MT system's output is then evaluated to see if this morphological contrast is preserved in the target language.

The authors extended this to three new language pairs: English-German, English-Finnish, and Turkish-English. They describe the process of generating these contrast pairs, which involves:

  1. Source Feature Identification: Identifying specific morphological features of interest in the source language.
  2. Variant Generation: Creating modified source sentences that differ only in the target feature.
  3. Language Model Filtering: Using language models to filter out less plausible sentence pairs.
  4. Scoring: Developing automatic scoring procedures that analyze the target language translations for the presence or absence of the correct morphological feature.

They also introduced new contrast types, such as those for compounds in German and participles in Turkish, and adapted existing ones for coreference and verb position.

Datasets

The paper utilizes existing large monolingual corpora for generating test suites:

  • English: English News-2007 and 2008 corpora.
  • Turkish: SETIMES2 corpus.

The evaluation was performed on the official WMT 2018 news translation task test sets for the respective language pairs. The authors also mention using WordNet for hyponym generation and a phrase table derived from the OPUS parallel corpus for German compound analysis.

Key Insights and Ideas

  • Morphology Matters, Even for Fluent NMT: Despite impressive fluency, NMT systems still struggle with specific morphological challenges, especially in morphologically rich languages.
  • Global vs. Local Evaluation: Systems that perform well on overall quality metrics (like BLEU or human evaluation) don't always excel in these fine-grained morphological tests, and vice-versa. This highlights the limitations of global metrics.
  • Language-Specific Challenges: Different languages present unique morphological hurdles. For instance, Finnish's lack of determiners and agglutinative nature create distinct challenges compared to German or Czech.
  • Rule-Based Systems Shine in Morphology: In some language pairs (e.g., English-Finnish), rule-based systems (RBMT) often outperform NMT systems on morphological tasks, likely due to their explicit handling of linguistic rules.
  • Consistency vs. Quality: Features designed to measure consistency (e.g., ensuring a word is translated the same way in similar contexts) and stability (e.g., identical translations for minor source changes) showed a weak correlation with overall translation quality, suggesting that robustness doesn't always equate to better translation.
  • Subword Units are Promising for Rare Words: The paper hints that systems using byte-pair encoding for subword units perform well on rare word features, suggesting this approach is effective for handling out-of-vocabulary words.

Key Results

The paper presents extensive tables of accuracy scores for various systems across different morphological features and language pairs. Some highlights include:

  • English-Czech: Transformer models (CUNI-Transformer) showed slightly lower accuracy than RNN models (uedin) on verb tasks and noun number, suggesting potential difficulties in conveying morphological features.
  • English-German: Tasks like verb negation, pronoun plural, and adjective superlatives were found to be relatively easy for most systems. Compound generation, however, proved very difficult, with accuracies ranging from 18.9% to 66.4%.
  • English-Finnish: Systems that explicitly model target morphology (HY-NMT2step, HY-AH) showed an advantage in paradigm completion features. The rule-based HY-AH system, despite its overall good performance, failed on subordinate clause types.
  • Turkish-English: Systems performed poorly on participle contrast pairs, with significant variation between systems.

Limitations and Future Work

  • Manual Evaluation is Crucial: While automatic scoring is used, the authors acknowledge the need for manual analysis to ensure the reliability of test suites and scoring procedures, especially for features where automatic evaluation might be ambiguous or prone to false positives/negatives.
  • Correlation with Overall Quality: The study found that performance on consistency and stability features doesn't strongly correlate with general translation quality, suggesting these metrics might be useful for specific applications (like post-editing) but not as a sole indicator of overall MT performance.
  • Idiomatic Expressions: The generation of test pairs had to carefully exclude idiomatic expressions, which can be challenging to handle and might not reflect typical translation scenarios.
  • Future Work: The authors suggest that the Morpheval framework is adaptable to new language pairs and features, paving the way for more in-depth analysis of MT system architectures, training data, and parameters. They also point to the need for further research into the correlation between morphological competence and overall translation quality.

Summary updated

BibTeX
@inproceedings{burlot2018wmt,
  author = {Franck Burlot and Yves Scherrer and Vinit Ravishankar and Ondřej Bojar and Stig-Arne Grönroos and Maarit Koponen and Tommi Nieminen and François Yvon},
  title = {{The WMT’18 Morpheval test suites for English-Czech, English-German, English-Finnish and Turkish-English}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure