TL;DR
Standard automated machine translation metrics evaluate translations in isolation, ignoring the comparative context that human raters naturally use. This paper introduces COMET-poly, featuring two new metric variants (COMET_poly-cand and COMET_poly-ic) that leverage alternative translation candidates or retrieved human-scored examples to significantly improve evaluation accuracy while running vastly faster than large language model judges.
What Problem Does This Paper Address?
Automated metrics for machine translation quality estimation (QE) typically look at just a source sentence and a single translation hypothesis. Humans, however, don't evaluate in a vacuum. When human raters judge translations, they look at multiple options side-by-side or calibrate their judgments based on previous examples they've seen (known as the sequence effect).
Evaluating a single translation in isolation makes automated metrics brittle, causing them to struggle with out-of-domain data or fail to detect subtle translation errors. The authors aim to bridge this gap by introducing contextual grounding into automated evaluation.
Methods
The authors propose two extensions to the popular COMET framework:
COMET_poly-cand: This model evaluates the target translation alongside alternative translation candidates of the same source sentence. It concatenates feature vectors from these multiple candidates, allowing the model to compare and contrast options.COMET_poly-ic: Inspired by retrieval-based in-context learning, this variant retrieves similar source-translation-score triplets from a database of historical human evaluations. By conditioning on these retrieved examples, the model adapts on the fly to specific domains or error patterns.
An overview of these architectures is illustrated below:
Datasets
- Training: Direct assessment scores from the WMT shared tasks up to 2023, totaling roughly 600,000 segments.
- Evaluation: The WMT 2024 metrics dataset (105 segments across 11 language pairs), evaluated using the Expert Error Span Annotation (ESA) protocol.
- Domain Adaptation: The BioMQM dataset, containing biomedical translations, was used to test the metrics' ability to handle out-of-domain data via in-context retrieval.
Key Insights and Ideas
- Context is King: Providing even a single extra translation candidate or retrieved example dramatically improves metric correlation with human judgments.
- Bad Candidates are Still Good Context: Surprisingly, the quality of the additional translation candidate in
COMET_poly-canddoesn't matter much. Even low-quality alternatives help the model map out "complementary error patterns," making it easier to spot flaws like undertranslation, literal copying, or refusals. - David Beats Goliath: A small, specialized model (~0.5B parameters) explicitly trained to use context can match or outperform a massive 70B parameter Large Language Model (LLM) evaluator like GEMBA, proving that architectural inductive biases can trump raw scale.
Key Results
- In reference-less quality estimation, adding just one additional candidate (
COMET_poly-cand) boosted Kendall's $\tau_b$ rank correlation from 0.079 (standard COMET) to 0.127, and Pearson correlation $\rho$ from 0.105 to 0.160. COMET_poly-icachieved a similar boost, raising $\tau_b$ to 0.116 using retrieved examples.- The proposed models outperformed the 70B parameter GEMBA baseline in multiple reference-less configurations.
- Efficiency:
COMET_poly-candprocesses 1,000 scores in 6.9 seconds on a single NVIDIA H100, compared to 254 seconds for the 4-bit quantized GEMBA-70B model—a massive win for practical deployment.
Limitations and Future Work
COMET_poly-candis inherently limited to scenarios where multiple translation hypotheses are available simultaneously (such as system benchmarking or reranking pipelines). It cannot evaluate a single standalone translation in isolation.- Both models are dependent on the quality and potential biases of the historical human-annotated data used in their training sets or retrieval knowledge bases. If the retrieval pool contains subpar annotations, metric performance may degrade.
- Future work could explore more sophisticated retrieval mechanisms or test the framework on a wider variety of specialized domains.
Summary updated