COMET-poly: Machine Translation Metric Grounded in Other Candidates

Maike Züfle, Vilém Zouhar, Tu Anh Dinh, Felipe Maia Polo, Jan Niehues, Mrinmaya Sachan

Abstract

Standard machine translation (MT) evaluation metrics typically operate in an "episodic" manner, assessing a single hypothesis translation in isolation relative to a source sentence or reference. This approach diverges from human evaluation processes, where raters often compare multiple outputs to calibrate their judgments. This paper introduces COMET-poly, a framework that incorporates comparative context into automated evaluation. By leveraging either multiple candidate translations for the same source (COMET-poly-cand) or retrieved examples from a labeled database (COMET-poly-ic), these models significantly outperform standard COMET and large language model (LLM)-based evaluators while maintaining high computational efficiency.

Introduction

In the field of Machine Translation, automated metrics like COMET have become the standard for assessing quality. However, these metrics usually receive a source sentence and a single hypothesis, producing a score without broader context. This "episodic" nature can lead to inconsistent scoring, particularly when the model encounters out-of-domain data where its internal quality scale may not be well-calibrated.

Human evaluation naturally avoids this pitfall through comparison. When humans rate translations, they often view several outputs side-by-side, allowing them to identify subtle nuances and errors that might be missed in isolation. COMET-poly aims to bridge this gap by providing automated metrics with a similar "big picture" perspective, using additional candidates or in-context examples as anchors for evaluation.

Methodology

The COMET-poly framework introduces two distinct methods for augmenting the evaluation context, both built upon a RoBERTa-large backbone.

1. COMET-poly-cand (Multiple Candidates)

This variant evaluates a target translation alongside $n$ additional candidate translations of the same source sentence. By processing these alternatives, the model can "compare and contrast" the target against other possible outputs. This comparative mechanism helps the model better identify superior translations. To ensure practical utility, the model can be trained to predict scores for all candidates simultaneously, minimizing the computational overhead of multiple passes.

2. COMET-poly-ic (In-Context Learning)

Drawing inspiration from the few-shot prompting capabilities of LLMs, COMET-poly-ic utilizes a retrieval-based approach. For a given input, the model retrieves "triplets"—consisting of a source, a translation, and a corresponding human score—from a large database of previously labeled data. These retrieved examples serve as grounding points, allowing the model to adapt its scoring logic to specific domains or linguistic styles dynamically without the need for fine-tuning.

Technically, both models augment the standard COMET feature vector. Rather than simply concatenating source and target embeddings, they append embeddings and difference vectors derived from the additional context (either the alternative candidates or the retrieved triplets).

Experimental Setup

The researchers trained the models on 600,000 segments of WMT (World Intellectual Property Organization) direct assessment data spanning up to 2023. Evaluation was conducted across two primary benchmarks:

  • WMT 2024 Test Set: Comprising 105,000 segments across 11 language pairs.
  • BioMQM Dataset: A specialized biomedical dataset used to test the models' robustness and domain adaptation capabilities in a high-stakes, technical field.

Key Insights

The "Pseudo-Reference" Effect

A significant finding of the study is that the additional candidates in the poly-cand setup do not need to be high-quality to be beneficial. Even mediocre alternative translations act as "pseudo-references," helping the model detect specific failure modes such as undertranslation (where the system merely copies the source) or refusals (where the system generates a canned response like "I cannot translate this").

Context vs. Scale

Experimental results demonstrate that COMET-poly-cand outperforms GEMBA, a metric based on the Llama 3.3 (70B) model. This suggests that providing specific, relevant context to a specialized, smaller model (RoBERTa-large) is more effective for evaluation tasks than relying on the massive parameter count of a general-purpose LLM in a zero-shot setting.

Domain Calibration via Retrieval

COMET-poly-ic proved particularly effective for domain adaptation. In the biomedical domain (BioMQM), the model's Mean Absolute Error (MAE) dropped significantly when it was allowed to retrieve relevant medical translation examples. This indicates that retrieval-based grounding can effectively calibrate a metric for specialized domains without requiring expensive retraining.

Performance and Efficiency

COMET-poly demonstrates substantial improvements in both accuracy and speed:

  • Accuracy: In reference-less Quality Estimation (QE) settings, adding a single extra candidate improved segment-level performance (Kendall’s tau-b) from 0.079 to 0.118.
  • Speed: COMET-poly variants are 40x to 70x faster than LLM-based evaluators. While GEMBA required nearly 200 seconds to score 1,000 segments, the COMET-poly models completed the same task in under 7 seconds on identical hardware.
  • Scalability: The joint prediction approach allows the model to score multiple candidates at once with negligible loss in accuracy, making it highly viable for large-scale system benchmarking.

Conclusion and Future Work

COMET-poly represents a shift from isolated to contextual machine translation evaluation. By grounding judgments in other candidates or historical data, the framework achieves state-of-the-art performance with a fraction of the parameters required by LLM-based metrics.

However, limitations remain. COMET-poly-cand requires multiple translations to be available, which may not always be the case in production environments. Additionally, the performance of COMET-poly-ic is contingent on the quality of the retrieval database. Future research will focus on more sophisticated retrieval mechanisms and the application of "poly" evaluation techniques to other generative tasks, such as text summarization and dialogue systems.

Summary updated

BibTeX
@misc{zufle2025comet,
  author = {Maike Züfle and Vilém Zouhar and Tu Anh Dinh and Felipe Maia Polo and Jan Niehues and Mrinmaya Sachan},
  title = {{COMET-poly: Machine Translation Metric Grounded in Other Candidates}},
  year = {2025},
  eprint = {2508.18549},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.18549},
}
Figure