The Word Sense Disambiguation Test Suite at WMT18

Annette Rios, Mathias Müller, Rico SennrichEMNLP 2018

TL;DR

This paper introduces a new, semi-automatic evaluation task for machine translation (MT) systems focused on accurately translating ambiguous words based on context. They found that while overall MT performance has improved significantly, unsupervised systems struggle with word sense disambiguation, and neural architectures are key to better lexical choice.

What Problem Does This Paper Address?

Translating ambiguous words correctly is a significant challenge for machine translation (MT) systems. A single word can have multiple meanings, and choosing the wrong one can lead to nonsensical or incomprehensible translations. Traditional MT evaluation metrics like BLEU, which assess overall sentence similarity to a reference, are too coarse-grained to capture these specific lexical choice errors. While previous work has shown neural MT (NMT) systems are more fluent, their adequacy (semantic faithfulness) has been more mixed, with mistranslations being a frequent error type. This paper aims to quantify and track progress in word sense disambiguation (WSD) for MT systems in a reproducible and semi-automatic manner.

Methods

The authors adapted the German-English Word Sense Disambiguation (WSD) test set, ContraWSD, to create a new evaluation task. They filtered the original set to reduce noise, specifically removing German words where multiple English translations covered multiple senses (e.g., "Stelle" translating to both "job" and "place" as "position"). The evaluation protocol was changed from scoring existing translations to assessing the MT system's direct output. This involves automatically checking if the system's translation of an ambiguous word uses the correct sense based on the context. If the output contains both correct and incorrect senses, or none of the known translations, manual inspection is triggered. They also developed a manual evaluation protocol for these edge cases.

Datasets

The primary dataset used is a filtered version of the German-English Word Sense Disambiguation (WSD) test set called ContraWSD (Rios Gonzales et al., 2017). This set was augmented with sentence pairs extracted from various parallel corpora, including WMT test and development sets, Crédit Suisse News Corpus, and corpora from OPUS (Global Voices, Books, EU Bookshop Corpus), and MultiUN.

Key Insights and Ideas

  • WSD is a critical component of MT adequacy: Errors in lexical choice for ambiguous words can severely degrade translation quality, even if fluency is high.
  • Neural architectures are crucial for lexical choice: The paper highlights a significant improvement in WSD accuracy with the adoption of Transformer and deep RNN architectures, suggesting that model architecture plays a vital role in understanding context for word sense.
  • Unsupervised systems lag significantly: Unsupervised MT systems demonstrate a clear disadvantage in WSD, often defaulting to a single translation for an ambiguous word regardless of context, leading to more frequent mistranslations.
  • Progress in WSD is measurable: The study shows a marked improvement in WSD accuracy from 2016 to 2018, indicating that MT systems are getting better at handling ambiguous words.
  • BLEU and WSD accuracy are correlated but distinct: While there's a strong correlation between WSD accuracy and BLEU on the WSD test suite, differences exist, with some systems performing better on WSD than their BLEU scores might suggest, possibly due to domain adaptation or training data differences.

Key Results

  • Significant Improvement in WSD Accuracy: The WSD accuracy on the test suite improved from 81% in 2016 to 93% in 2018 for the best systems.
  • NMT vs. SMT: Neural MT systems (e.g., uedin-nmt-2017) achieved higher WSD accuracy (84.9% automatic) compared to syntax-based SMT systems (e.g., uedin-syntax-2016 at 79.7% automatic).
  • Unsupervised Systems Performance: Unsupervised systems like RWTH-UNSUPER and LMU-unsup performed poorly, with WSD accuracies around 42-47%.
  • Error Analysis: The error analysis revealed that while predicting the wrong sense was common, unsupervised systems also had a higher proportion of untranslated ambiguous words.
  • Architecture Matters: Deep Transformer and RNN architectures generally outperformed shallow RNNs in WSD accuracy.

Limitations and Future Work

  • Limited Scope of Ambiguous Words: The current study focuses on a small set of 20 ambiguous nouns and 45 word senses. The authors suggest evaluating WSD on a larger set of words, including verbs and adverbs, which constitute a substantial proportion of lexical choice errors in previous analyses.
  • Semi-Automatic Evaluation: While mostly automatic, about 5% of cases required manual inspection, indicating potential for further refinement of the automatic evaluation protocol.
  • Domain Specificity: The authors note that some systems might perform better due to domain focus or training data differences, and that online systems might not be subject to the same constraints as shared task systems. Future work could explore these factors more deeply.

Summary updated

BibTeX
@inproceedings{rios2018word,
  author = {Annette Rios and Mathias Müller and Rico Sennrich},
  title = {{The Word Sense Disambiguation Test Suite at WMT18}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure