An Awkward Disparity between BLEU / RIBES Scores and Human Judgements in Machine Translation

Liling Tan, Jon Dehdari, Josef van GenabithWAT 2015 · OpenReview

Abstract

Automatic metrics like BLEU and RIBES are the standard for evaluating machine translation (MT) systems. However, this paper presents a "reality check" for MT researchers by demonstrating a case where these metrics fail to align with human perception. In a Korean-to-Japanese patent translation task from the Workshop on Asian Translation (WAT 2015), the authors' system achieved a massive 16-point lead in BLEU score over a baseline, yet was rated significantly worse by human evaluators. This study explores the causes of this "awkward disparity," focusing on the limitations of surface-level string matching in technical domains.

Introduction

In Machine Translation, progress is typically measured using automatic metrics. BLEU (Bilingual Evaluation Understudy) measures n-gram overlap between machine output and human references, while RIBES focuses on word-order differences, a critical factor for language pairs like Korean and Japanese. While a higher BLEU score is generally equated with better translation quality, researchers have long suspected a "schism" between these scores and actual human judgment. This paper provides a concrete example of this divergence in a real-world shared task.

Methodology and Datasets

The authors performed a meta-evaluation of their phrase-based Statistical Machine Translation (SMT) system (Team ID: Sense) against the official WAT 2015 baseline. The study utilized the JPO Patent Corpus, consisting of 1 million training sentences across chemistry, electricity, mechanical engineering, and physics domains.

The authors' system was built using the Moses toolkit with several optimized parameters compared to the organizers' baseline, as detailed in the table below:

Comparison of system parameters between the organizers' baseline and the authors' replica.

Key optimizations included advanced tokenization (KoNLPy for Korean, MeCab for Japanese), a significantly higher distortion limit (20 vs. 0), and increased input document length. Human judgment scores were calculated via pairwise comparisons by five crowd-sourced evaluators.

The "Awkward Disparity" in Results

The quantitative results reveal a startling contradiction between automatic metrics and human evaluation.

Comparison of RIBES, BLEU, and Human scores across different system versions.

As shown in the results table, the authors' "MERT 2" system outperformed the baseline by 16.01 BLEU points. In most MT research, such a margin would be considered a massive breakthrough. However, human evaluators gave this same system a score of -17.75, indicating it was perceived as significantly worse than the baseline.

Analysis of the Metric-Human Gap

The authors identify several reasons for this failure of automatic metrics:

  • The n-gram "Multiplier Effect": BLEU uses a geometric average of n-gram precisions. A single lexical change can break multiple overlapping n-grams, causing a score to drop sharply even if the change is semantically neutral or preferred by humans.
  • Adequacy vs. Precision: In technical patent translation, accuracy is paramount. A system might swap two numerical values (e.g., temperatures), which BLEU treats as a minor reordering error. To a human, this is a critical "adequacy error" that makes the translation useless or dangerous.
  • The "Fluency" Trap: Metrics often reward output that "looks" like the reference (fluency) over output that conveys the correct meaning (adequacy). A fluent but factually incorrect sentence can still achieve a high BLEU score.
  • Crowd-Sourcing Limitations: Patent translation is highly specialized. Non-expert crowd-sourced evaluators might prefer natural-sounding language, whereas automatic metrics and reference translations might adhere to rigid, domain-specific terminology.

Conclusion

This study reinforces that while automatic metrics are essential for rapid development, they are not a substitute for human evaluation, especially in specialized domains. The "awkward disparity" observed here highlights the need for more robust metrics that account for semantic adequacy and content recall rather than just surface-level string matching.

Summary updated

BibTeX
@inproceedings{tan2015awkward,
  author = {Liling Tan and Jon Dehdari and Josef van Genabith},
  title = {{An Awkward Disparity between BLEU / RIBES Scores and Human Judgements in Machine Translation}},
  booktitle = {WAT},
  year = {2015},
  url = {https://openreview.net/forum?id=flSKOo9Nong},
}
Figure