Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Papers

Benjamin Marie, Atsushi Fujita, Raphael RubinoACL 2021

TL;DR

This paper reveals a concerning trend in machine translation (MT) research: a growing reliance on superficial evaluation methods, primarily using BLEU scores without statistical rigor or human validation. The authors propose a guideline and scoring system to encourage more credible MT evaluations, aiming to improve the scientific rigor of the field.

What Problem Does This Paper Address?

The field of Machine Translation (MT) is rapidly advancing, with new methods and algorithms constantly being proposed. A crucial aspect of this research is evaluating the quality of translations produced by these systems. Traditionally, this has been done using automatic metrics like BLEU, and sometimes human evaluation. However, the authors observe that the way MT evaluations are conducted has become less rigorous over time. Specifically, they question whether comparisons between MT systems are truly valid, given the lack of standardized practices and the potential for superficial analysis. This paper aims to systematically investigate the state of MT evaluation practices in published research to identify and quantify these issues.

Methods

The core of this research is a large-scale meta-evaluation. The authors manually annotated 769 research papers published in *ACL conferences between 2010 and 2020 that compared at least two MT systems. For each paper, they extracted information on:

  • Automatic metrics used: Identifying all metrics employed, excluding those that only assess specific aspects of translation.
  • Human evaluation: Whether a human evaluation of translation quality was conducted.
  • Statistical significance testing: Whether statistical tests were performed to validate differences in metric scores.
  • Copied scores: Whether papers directly compared their results with scores reported in previous work.
  • SacreBLEU usage: Whether the SacreBLEU tool was used for standardized metric reporting.
  • Data comparability: If scores were copied from previous work, whether it was confirmed that the compared MT systems used identical training, validation, and testing data.

Based on these annotations, they analyzed trends and identified common pitfalls in MT evaluation. They also conducted their own experiments to demonstrate the impact of these pitfalls.

Datasets

The paper does not introduce new datasets. Instead, it analyzes existing research papers that utilize various datasets for MT evaluation. The authors also use data from the WMT20 news translation shared task for their own experimental demonstrations to illustrate the impact of data preprocessing on metric scores.

Key Insights and Ideas

  • The "99% BLEU" Phenomenon: An overwhelming majority (98.8%) of MT papers use BLEU scores for evaluation, and a significant portion (74.3%) rely solely on BLEU, often without any other metrics or human evaluation.
  • The Decline of Statistical Rigor: The use of statistical significance testing in MT evaluations has sharply decreased since 2016, meaning many papers draw conclusions from differences in scores without verifying if these differences are statistically meaningful.
  • The Rise of "Copied Scores": Nearly 40% of papers in 2019-2020 compare their results with scores reported in previous work, often without ensuring comparability.
  • SacreBLEU's Slow Adoption: Despite being a crucial tool for standardizing metric reporting and ensuring comparability, SacreBLEU is still not widely adopted by the MT community.
  • Data Inconsistency is Rampant: A significant portion of papers (38.5% in 2019-2020) compare MT systems that were trained, validated, or tested on different datasets or with different preprocessing steps, making direct comparisons unreliable.
  • Accumulation of Pitfalls: The paper highlights how these individual issues accumulate, leading to "scientifically dubious" evaluations where claims of superiority are not well-supported.

Key Results

  • Over-reliance on BLEU: 98.8% of papers use BLEU, and 74.3% use only BLEU. This rises to 82.1% for papers from 2019-2020.
  • Decreasing Statistical Significance Testing: The percentage of papers performing statistical significance testing has dropped significantly since 2016.
  • Prevalence of Copied Scores: By 2019-2020, around 40% of papers copied scores from previous work.
  • Data Inconsistency: 38.5% of papers in 2019-2020 compared systems that used different datasets or preprocessing.
  • Worsening Evaluation Trends: The average meta-evaluation score (based on their proposed guideline) has worsened over the years, indicating a decline in evaluation quality.

Limitations and Future Work

Limitations:

  • The study is limited to *ACL conferences, potentially missing relevant MT research published elsewhere.
  • The meta-evaluation scoring method proposed is a simplified approach and does not guarantee a "flawless" evaluation, but rather a "better" one.
  • The paper acknowledges that some papers might perform significance testing without mentioning it, leading to potential underestimation in their annotations.

Future Work:

  • The authors plan to extend their meta-evaluation to publications in other research domains like Machine Learning and Artificial Intelligence.
  • They encourage NLP researchers to conduct similar meta-evaluations in their respective areas to identify and address concerning trends.
  • They propose a guideline and scoring method for automatic MT evaluation, hoping for its adoption by authors and reviewers to improve the scientific credibility of MT research.

Summary updated

BibTeX
@inproceedings{marie2021scientific,
  author = {Benjamin Marie and Atsushi Fujita and Raphael Rubino},
  title = {{Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Papers}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure