Beyond Reference: Evaluating High Quality Translations Better than Human References

Keonwoong Noh, Seokjin Oh, Woohwan JungEMNLP 2024 · DOI

Introduction

Traditional Machine Translation (MT) evaluation has long relied on the assumption that human-written references represent an absolute "ceiling" of quality. Metrics such as BLEU, BERTScore, and COMET typically measure the similarity of a machine-generated candidate to this gold standard. However, as Large Language Models (LLMs) continue to advance, they frequently produce translations that surpass human references in terms of fluency, nuance, and accuracy.

Existing metrics suffer from reference bias, meaning they penalize any deviation from the reference, even if the deviation is an improvement. This creates a bottleneck in MT research: we cannot accurately rank top-tier systems if our evaluation tools are capped at "human-level." This paper introduces RESUME (Residual score Metric), a novel approach that enables the identification and scoring of translations that outperform their human references.

The RESUME Framework

RESUME is designed as a complementary metric that can be added to existing absolute MT metrics. Unlike traditional scores, RESUME outputs a relative value (a residual) between -1 and 1.

  1. Relative Quality: If a candidate $c$ is superior to the reference $r$, RESUME assigns a positive score. If it is inferior, the score is negative.
  2. Integrated Scoring: The final evaluation is a combination of a base metric and the residual: $$\text{Score} = \text{Absolute Metric Score} + \lambda \cdot \text{RESUME}(s, r, c)$$ where $s$ is the source text and $\lambda$ is a scaling hyperparameter.

Training via the "Swapping" Heuristic

A significant hurdle in developing such a metric is the lack of datasets where machine outputs are explicitly labeled as "better than human." To overcome this, the authors employed a clever data augmentation strategy. By taking existing human-rated datasets and swapping the roles of the reference and the candidate, they forced the model to treat the (higher-quality) human reference as a "candidate" and the (lower-quality) machine output as a "reference." This taught the model to recognize the linguistic features of superior quality even when the original labels were bounded.

Experimental Results

Improved Correlation with Human Judgment

The effectiveness of RESUME was evaluated on the WMT 22 MQM dataset using expert-level annotations. The results demonstrate that RESUME consistently enhances the alignment between automatic metrics and human expert judgments (measured by Kendall’s $\tau$) across a wide variety of supervised and unsupervised baselines.

RESUME consistently improves the Kendall’s τ correlation across various MT metrics, including UniTE, COMET, and BLEU.

Overcoming Reference Bias

To specifically test for reference bias, the authors utilized a post-editing dataset where experts refined initial translations. In these cases, the post-edited version is objectively better than the original reference. Traditional metrics failed significantly in this test, often scoring the original reference higher than the improved version. RESUME, however, showed a massive leap in sensitivity to these improvements.

Comparison of metrics in identifying post-edited translations as superior to original references. RESUME correctly identifies the improvement 59% of the time, compared to less than 7% for standard metrics.

Key Insights and Implications

  • LLM Performance: The application of RESUME reveals that traditional metrics have been systematically underestimating LLMs. For instance, when RESUME is applied, GPT-4’s rankings jump significantly, suggesting that its translations are often more accurate than the human references provided in standard benchmarks.
  • Robustness to Flawed References: Case studies indicated that RESUME can correctly identify superior translations even when human references contain typos, omissions, or "translationese," whereas similarity-based metrics would penalize the model for these correct deviations.

Conclusion

RESUME represents a shift from "similarity-to-reference" to "relative-quality-assessment." By breaking the artificial ceiling imposed by human references, it provides a more accurate and sensitive tool for evaluating the next generation of high-quality translation systems. While it introduces some computational overhead, its ability to mitigate reference bias makes it a vital addition to the MT evaluation toolkit.

Summary updated

BibTeX
@inproceedings{noh2024beyond,
  author = {Keonwoong Noh and Seokjin Oh and Woohwan Jung},
  title = {{Beyond Reference: Evaluating High Quality Translations Better than Human References}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.294},
}
Figure