TL;DR
This paper highlights a critical inconsistency in how BLEU scores are reported in machine translation research, making it difficult to compare results across studies. The authors propose a standardized approach using metric-internal reference processing and introduce a new tool, SACREBLEU, to enforce this standard and ensure reproducible BLEU scores.
What Problem Does This Paper Address?
The field of machine translation heavily relies on the BLEU (Bilingual Evaluation Understudy) score to measure the quality of translations. However, BLEU is not a single, fixed metric but rather a parameterized one. Crucially, the way input text and reference translations are tokenized and preprocessed before being fed into the BLEU calculation can drastically alter the resulting score. This variability, coupled with the common practice of researchers not fully reporting their preprocessing steps, means that BLEU scores reported in different papers are often not directly comparable. This hinders scientific progress by making it difficult to replicate results and accurately assess the impact of new methods. The paper aims to bring attention to this "under-recognized problem" and offer a solution.
Methods
The paper's core contribution is not a new algorithm for calculating BLEU itself, but rather a proposal for standardizing its reporting. The authors advocate for a strict separation between user-supplied preprocessing and metric-internal processing. Specifically, they propose that researchers should only use the metric-internal tokenization and preprocessing for reference translations. This ensures that the same reference data is processed identically for all systems being compared.
To facilitate this standardization, the authors introduce SACREBLEU, a Python package. SACREBLEU automates the process of downloading and storing standard test sets and their references, effectively creating a "protective layer" between the user and the raw reference data. It also enforces metric-internal preprocessing and provides a version string that documents the exact settings used for scoring, promoting transparency and reproducibility.
Datasets
The paper uses data from the WMT (Workshop on Machine Translation) evaluations, specifically mentioning the WMT'14 English-German track. It also references the IWSLT (International Workshop on Spoken Language Translation) test sets. The authors highlight a specific issue with the WMT'14 EN-DE dataset, noting that there are two versions (one with 2,737 sentences and another with 3,004 sentences) and that researchers often fail to specify which they used, contributing to score variance. SACREBLEU is designed to handle these standard test sets.
Key Insights and Ideas
- BLEU is not one metric, but many: The paper emphasizes that BLEU's score is highly dependent on its parameters (e.g., number of references, n-gram length, smoothing) and, critically, on how the reference text is tokenized and preprocessed.
- Preprocessing is a major culprit: The authors identify user-supplied reference tokenization and normalization as the primary reason for incomparable BLEU scores. Differences as large as 1.8 points can arise from seemingly minor variations in preprocessing.
- Reproducibility is suffering: The lack of clear reporting on preprocessing steps makes it incredibly difficult for other researchers to replicate results or even understand the context of reported scores.
- Analogy to Parsing: The paper draws a parallel to the field of English parsing, where the adoption of a standard codebase (evalb) for computing F1 scores has enabled decades of comparable results.
- A call for standardization: The authors argue that the machine translation community needs to adopt a similar standardized approach to BLEU reporting.
Key Results
The paper quantifies the impact of different preprocessing schemes on BLEU scores. Table 1 demonstrates that variations in tokenization (basic, split, unk, metric) can lead to differences of up to 1.8 BLEU points for a single language arc. The average difference across schemes is around 1.0 point. The paper also shows (in Table 2) that even for influential papers, it can be difficult to determine which BLEU configuration was used, highlighting the severity of the reporting problem. The introduction of SACREBLEU is presented as a practical solution to enforce standardized scoring.
Limitations and Future Work
The authors acknowledge that their proposed solution requires community adoption. While they recommend the WMT standard processing scheme for backwards compatibility, the ultimate success of SACREBLEU depends on researchers choosing to use it. Future work could involve expanding SACREBLEU to support more test sets and metrics, and further encouraging community adoption through workshops and publications. The paper also briefly touches on other dataset specification issues (like the WMT'14 EN-DE dataset versions) which could be further explored.
Summary updated