Evaluating Conjunction Disambiguation on English-to-German and French-to-German WMT 2019 Translation Hypotheses

Maja PopovićACL 2019

TL;DR

This paper tackles the challenge of accurately translating ambiguous conjunctions like "but" and "mais" into German, which have two distinct translations ("aber" and "sondern") depending on sentence structure. They introduce a new test set and evaluate WMT 2019 translation systems, finding that while "aber" is handled well, "sondern" remains a challenge, with multilingual Transformer models showing the most promise.

What Problem Does This Paper Address?

Neural Machine Translation (NMT) systems have made significant strides in producing fluent and grammatical translations. However, they still struggle with semantic faithfulness, particularly when dealing with ambiguous words. This paper focuses on a specific type of ambiguity: conjunctions. In German, the English conjunction "but" and its French equivalent "mais" can be translated into either "aber" or "sondern." The choice between these two German conjunctions is not arbitrary; it depends on the grammatical structure of the sentence. Specifically, "sondern" is used to express a contradiction after a negative clause, while "aber" is more general and can be used in various contexts, including after positive or negative clauses. Existing research has not extensively evaluated NMT systems' ability to handle this specific type of structural ambiguity in conjunction translation. This paper aims to fill that gap by providing a quantitative evaluation of how well state-of-the-art NMT systems disambiguate these conjunctions.

Methods

The authors developed a semi-automatic method for creating a test set to evaluate conjunction disambiguation. They extracted short segments from bilingual subtitle corpora, focusing on sentences containing the ambiguous conjunctions in English and French, and their potential German translations. Manual filtering was then applied to remove noisy or complex sentences. The evaluation itself is primarily automatic, focusing on the presence of the correct German conjunction in the translated output. For a small percentage of cases (around 1-2%) that are unclear, manual inspection is performed. The evaluation criteria are straightforward: a translation is considered correct if it uses the appropriate German conjunction ("aber" or "sondern") based on the source sentence's structure. Errors are categorized as using the "opposite" conjunction or other types of errors.

Datasets

The primary dataset used for creating the test sets was the OpenSubtitles corpus (version 2018). The test sets themselves were specifically constructed for this study and contain approximately 1000 sentences for each source language (English and French) targeting German. These test sets are distinct from standard NMT evaluation sets because they do not rely on reference translations for evaluation. Instead, the evaluation is based on the correctness of the conjunction choice, which can be determined without a reference. The paper also references the WMT 2019 News Commentary training corpus for statistics on the general frequency of these conjunction types.

Key Insights and Ideas

  • "Aber" is Easy, "Sondern" is Hard: All evaluated NMT systems performed exceptionally well in translating sentences that require "aber," achieving near-perfect accuracy. This suggests that NMT models have largely mastered this more common and less structurally constrained translation.
  • Multilingual Models Shine: The multilingual Transformer model from TartuNLP, trained on all WMT language pairs using the Latin script, consistently outperformed other systems for both English-to-German and French-to-German translation, particularly on the challenging "sondern" cases. This highlights the benefit of a unified model learning from diverse linguistic structures.
  • Synthetic Data's Ambiguous Role: The paper found no clear advantage or disadvantage for systems that used synthetic back-translated data versus those trained solely on natural parallel data. This suggests that while synthetic data can augment training sets, it doesn't inherently guarantee better performance on specific linguistic challenges like conjunction disambiguation.
  • Qualitative vs. Quantitative: Highly ranked systems not only correctly disambiguate conjunctions but also produce generally fluent and adequate translations. Conversely, lower-ranked systems tend to suffer from both poor conjunction choice and overall translation quality.

Key Results

  • "Aber" Accuracy: All systems achieved near 100% accuracy for the "aber" conjunction in both English-to-German and French-to-German translation.
  • "Sondern" Accuracy: For the "sondern" conjunction, system accuracies ranged from 78% to 97%. The most common error was substituting "sondern" with "aber."
  • Top Performer: The multilingual TartuNLP system achieved the highest "full" accuracy for "sondern" in both language pairs, with 97.3% for En-De and 96.9% for Fr-De.
  • Lowest Performer: The en-de-task system had the lowest "full" accuracy for "sondern" at 78.2%.

Limitations and Future Work

The authors acknowledge that their study is focused on a single ambiguous conjunction ("but"/"mais") and a limited set of target language pairs (English-to-German and French-to-German). They propose extending the test set to include more conjunctions and their variants, as well as exploring additional language pairs. They also suggest that a more systematic quantitative analysis of the correlation between conjunction disambiguation and overall translation adequacy/fluency would be valuable future work.

Summary updated

BibTeX
@inproceedings{popovic2019evaluating,
  author = {Maja Popović},
  title = {{Evaluating Conjunction Disambiguation on English-to-German and French-to-German WMT 2019 Translation Hypotheses}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure