EvalD Reference-Less Discourse Evaluation for WMT18

Ondřej Bojar, Jiří Mírovský, Kateřina Rysová, Magdaléna RysováEMNLP 2018

TL;DR

This paper explores using EVALD, a reference-less discourse evaluation tool, to assess machine translation (MT) quality beyond sentence-level. They found that while the native-speaker version of EVALD was ineffective for MT evaluation, the non-native speaker version showed promise in distinguishing between different MT systems, suggesting a potential avenue for evaluating document-level translation quality.

What Problem Does This Paper Address?

Machine translation (MT) has made incredible strides, with neural MT (NMT) systems often achieving human-level quality on individual sentences. However, evaluating MT at the document level, where discourse coherence and flow become crucial, remains a significant challenge. Traditional MT evaluation metrics like BLEU focus on word overlap with reference translations and struggle to capture these higher-level linguistic phenomena. This paper addresses the need for evaluation methods that can assess the quality of translated text as a whole, similar to how human readers evaluate written content, without relying on human-written reference translations.

Methods

The authors employ EVALD (Evaluator of Discourse), a tool originally designed to assess the quality of human-written texts for both native and non-native speakers. EVALD utilizes a set of 180 features, categorized into shallow features (spelling, morphology, syntax) and deep text features (coreference, discourse connectives, information structure). They specifically use two versions of EVALD: EVALD L1 (trained on native Czech essays) and EVALD L2 (trained on Czech essays by non-native speakers). The core idea is to adapt a human-text evaluation tool for MT evaluation. They hypothesize that EVALD L2, trained on less-than-perfect texts, might be more suitable for evaluating MT outputs, which can also exhibit discourse-level disruptions.

Datasets

The primary dataset used for evaluation is the Michigan Corpus of Upper-Level Student Papers (MICUSP). This corpus contains approximately 830 original English texts (2.6 million words) from four academic areas (Humanities and Arts, Social Sciences, Biological and Health Sciences, Physical Sciences) and various genres (argumentative essay, creative writing, critique, proposal, research paper). The authors selected 56 texts written by native English speakers and 51 by non-native speakers. These texts were then translated into Czech by various MT systems participating in the WMT18 News Translation Task. The source texts themselves were also used as a baseline for EVALD evaluation, representing the "worst possible" quality.

Key Insights and Ideas

  • EVALD L1 is not suitable for MT evaluation: The EVALD version trained on native speakers (L1) assigned very similar, high scores to all MT systems, indicating it was too lenient and not discerning enough for this task.
  • EVALD L2 shows promise for MT evaluation: The version trained on non-native speakers (L2) provided more varied scores, allowing for some differentiation between MT systems. This suggests that a tool trained on less-than-perfect human writing can be adapted to evaluate imperfect machine translations.
  • Nativeness of the original author matters: The evaluation scores showed the highest variance based on whether the original English text was written by a native or non-native speaker. This highlights the impact of source text quality on the perceived quality of the translation, even at the discourse level.
  • Discourse-specific features are key: Comparing EVALD scores using only deep discourse features versus all features showed that the discourse-specific features were the primary drivers of rank differences among MT systems, reinforcing their importance for evaluating coherence.

Key Results

The paper reports EVALD L2 scores for individual MT systems and across different genres and domains. While all MT systems received relatively high scores with EVALD L2 (suggesting overestimation of quality), some differences emerged. For instance, CUNI Transformer and UEDIN generally scored higher than CUNI Moses. The variance in scores was highest when considering the nativeness of the original author, with texts originally written by native speakers receiving higher EVALD scores. Creative writing texts, regardless of the MT system, received uniformly high scores, which the authors attribute to EVALD potentially detecting superficial discourse features that are present in these texts and preserved by the MT systems.

Limitations and Future Work

The authors acknowledge that EVALD L2, even with its promise, still overestimates the quality of MT outputs. They note that the current evaluation is limited by the lack of a contrastive manual evaluation of the translated texts. Future work should involve a more careful analysis of EVALD features and their adaptation to better discern MT quality. Direct comparison with manual assessments of translation quality is also recommended to establish EVALD as a robust MT evaluation method. The authors also suggest that further exploration of EVALD features and their adaptation will be needed to obtain more discerning evaluation.

Summary updated

BibTeX
@inproceedings{bojar2018evald,
  author = {Ondřej Bojar and Jiří Mírovský and Kateřina Rysová and Magdaléna Rysová},
  title = {{EvalD Reference-Less Discourse Evaluation for WMT18}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure