COMET-poly: Machine Translation Metric Grounded in Other Candidates

Maike Züfle, Vilém Zouhar, Tu Anh Dinh, Felipe Maia Polo, Jan Niehues, Mrinmaya Sachan

TL;DR

Standard automated machine translation metrics evaluate translations in isolation, ignoring the comparative context that human raters naturally use. This paper introduces COMET-poly, featuring two new metric variants (COMET_poly-cand and COMET_poly-ic) that leverage alternative translation candidates or retrieved human-scored examples to significantly improve evaluation accuracy while running vastly faster than large language model judges.

What Problem Does This Paper Address?

Automated metrics for machine translation quality estimation (QE) typically look at just a source sentence and a single translation hypothesis. Humans, however, don't evaluate in a vacuum. When human raters judge translations, they look at multiple options side-by-side or calibrate their judgments based on previous examples they've seen (known as the sequence effect).

Evaluating a single translation in isolation makes automated metrics brittle, causing them to struggle with out-of-domain data or fail to detect subtle translation errors. The authors aim to bridge this gap by introducing contextual grounding into automated evaluation.

Methods

The authors propose two extensions to the popular COMET framework:

  1. COMET_poly-cand: This model evaluates the target translation alongside alternative translation candidates of the same source sentence. It concatenates feature vectors from these multiple candidates, allowing the model to compare and contrast options.
  2. COMET_poly-ic: Inspired by retrieval-based in-context learning, this variant retrieves similar source-translation-score triplets from a database of historical human evaluations. By conditioning on these retrieved examples, the model adapts on the fly to specific domains or error patterns.

An overview of these architectures is illustrated below:

Figure 1: The COMETpoly-ic model consults a knowl- edge base of previously human-scored translations be- fore assigning the quality estimation score to the can- didate translation. The COMETpoly-cand considers other possible translations apart from the candidate one. Both metrics work better than just providing the source and the translation.

Datasets

  • Training: Direct assessment scores from the WMT shared tasks up to 2023, totaling roughly 600,000 segments.
  • Evaluation: The WMT 2024 metrics dataset (105 segments across 11 language pairs), evaluated using the Expert Error Span Annotation (ESA) protocol.
  • Domain Adaptation: The BioMQM dataset, containing biomedical translations, was used to test the metrics' ability to handle out-of-domain data via in-context retrieval.

Key Insights and Ideas

  • Context is King: Providing even a single extra translation candidate or retrieved example dramatically improves metric correlation with human judgments.
  • Bad Candidates are Still Good Context: Surprisingly, the quality of the additional translation candidate in COMET_poly-cand doesn't matter much. Even low-quality alternatives help the model map out "complementary error patterns," making it easier to spot flaws like undertranslation, literal copying, or refusals.
  • David Beats Goliath: A small, specialized model (~0.5B parameters) explicitly trained to use context can match or outperform a massive 70B parameter Large Language Model (LLM) evaluator like GEMBA, proving that architectural inductive biases can trump raw scale.

Key Results

  • In reference-less quality estimation, adding just one additional candidate (COMET_poly-cand) boosted Kendall's $\tau_b$ rank correlation from 0.079 (standard COMET) to 0.127, and Pearson correlation $\rho$ from 0.105 to 0.160.
  • COMET_poly-ic achieved a similar boost, raising $\tau_b$ to 0.116 using retrieved examples.
  • The proposed models outperformed the 70B parameter GEMBA baseline in multiple reference-less configurations.
  • Efficiency: COMET_poly-cand processes 1,000 scores in 6.9 seconds on a single NVIDIA H100, compared to 254 seconds for the 4-bit quantized GEMBA-70B model—a massive win for practical deployment.
Table 1: Results for COMETpoly-cand and COMETpoly-ic. The first row shows the standard COMET. The middle and bottom parts show that adding additional translation candidates and in-context examples boosts performance.

Limitations and Future Work

  • COMET_poly-cand is inherently limited to scenarios where multiple translation hypotheses are available simultaneously (such as system benchmarking or reranking pipelines). It cannot evaluate a single standalone translation in isolation.
  • Both models are dependent on the quality and potential biases of the historical human-annotated data used in their training sets or retrieval knowledge bases. If the retrieval pool contains subpar annotations, metric performance may degrade.
  • Future work could explore more sophisticated retrieval mechanisms or test the framework on a wider variety of specialized domains.

Summary updated

Abstract

Automated metrics for machine translation attempt to replicate human judgment. Unlike humans, who often assess a translation in the context of multiple alternatives, these metrics typically consider only the source sentence and a single translation. This discrepancy in the evaluation setup may negatively impact the performance of automated metrics. We propose two automated metrics that incorporate additional information beyond the single translation. COMET-polycand uses alternative translations of the same source sentence to compare and contrast with the translation at hand, thereby providing a more informed assessment of its quality. COMET-polyic, inspired by retrieval-based in-context learning, takes in translations of similar source texts along with their human-labeled quality scores to guide the evaluation. We find that including a single additional translation in COMET-polycand improves the segment-level metric performance (0.079 to 0.118 Kendall's tau-b correlation), with further gains when more translations are added. Incorporating retrieved examples in COMET-polyic yields similar improvements (0.079 to 0.116 Kendall's tau-b correlation). We release our models publicly.
BibTeX
@misc{zufle2025comet,
  author = {Maike Züfle and Vilém Zouhar and Tu Anh Dinh and Felipe Maia Polo and Jan Niehues and Mrinmaya Sachan},
  title = {{COMET-poly: Machine Translation Metric Grounded in Other Candidates}},
  year = {2025},
  eprint = {2508.18549},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.18549},
}
Figure