Introduction
Traditional Machine Translation (MT) evaluation has long relied on the assumption that human-written references represent an absolute "ceiling" of quality. Metrics such as BLEU, BERTScore, and COMET typically measure the similarity of a machine-generated candidate to this gold standard. However, as Large Language Models (LLMs) continue to advance, they frequently produce translations that surpass human references in terms of fluency, nuance, and accuracy.
Existing metrics suffer from reference bias, meaning they penalize any deviation from the reference, even if the deviation is an improvement. This creates a bottleneck in MT research: we cannot accurately rank top-tier systems if our evaluation tools are capped at "human-level." This paper introduces RESUME (Residual score Metric), a novel approach that enables the identification and scoring of translations that outperform their human references.
The RESUME Framework
RESUME is designed as a complementary metric that can be added to existing absolute MT metrics. Unlike traditional scores, RESUME outputs a relative value (a residual) between -1 and 1.
- Relative Quality: If a candidate $c$ is superior to the reference $r$, RESUME assigns a positive score. If it is inferior, the score is negative.
- Integrated Scoring: The final evaluation is a combination of a base metric and the residual: $$\text{Score} = \text{Absolute Metric Score} + \lambda \cdot \text{RESUME}(s, r, c)$$ where $s$ is the source text and $\lambda$ is a scaling hyperparameter.
Training via the "Swapping" Heuristic
A significant hurdle in developing such a metric is the lack of datasets where machine outputs are explicitly labeled as "better than human." To overcome this, the authors employed a clever data augmentation strategy. By taking existing human-rated datasets and swapping the roles of the reference and the candidate, they forced the model to treat the (higher-quality) human reference as a "candidate" and the (lower-quality) machine output as a "reference." This taught the model to recognize the linguistic features of superior quality even when the original labels were bounded.
Experimental Results
Improved Correlation with Human Judgment
The effectiveness of RESUME was evaluated on the WMT 22 MQM dataset using expert-level annotations. The results demonstrate that RESUME consistently enhances the alignment between automatic metrics and human expert judgments (measured by Kendall’s $\tau$) across a wide variety of supervised and unsupervised baselines.
Overcoming Reference Bias
To specifically test for reference bias, the authors utilized a post-editing dataset where experts refined initial translations. In these cases, the post-edited version is objectively better than the original reference. Traditional metrics failed significantly in this test, often scoring the original reference higher than the improved version. RESUME, however, showed a massive leap in sensitivity to these improvements.
Key Insights and Implications
- LLM Performance: The application of RESUME reveals that traditional metrics have been systematically underestimating LLMs. For instance, when RESUME is applied, GPT-4’s rankings jump significantly, suggesting that its translations are often more accurate than the human references provided in standard benchmarks.
- Robustness to Flawed References: Case studies indicated that RESUME can correctly identify superior translations even when human references contain typos, omissions, or "translationese," whereas similarity-based metrics would penalize the model for these correct deviations.
Conclusion
RESUME represents a shift from "similarity-to-reference" to "relative-quality-assessment." By breaking the artificial ceiling imposed by human references, it provides a more accurate and sensitive tool for evaluating the next generation of high-quality translation systems. While it introduces some computational overhead, its ability to mitigate reference bias makes it a vital addition to the MT evaluation toolkit.
Summary updated