TransEvalnia: Reasoning-based Evaluation and Ranking of Translations

Richard Sproat, Tianyu Zhao, Llion JonesarXiv 2025 · DOI · arXiv

TransEvalnia is a new framework that moves machine translation evaluation away from "black-box" numerical scores toward detailed, human-like reasoning. By using a novel "three-step interleaving" prompt strategy, it significantly reduces the common problem of position bias in LLMs while providing fine-grained feedback that professional translators find highly accurate.

The Challenge of Modern MT Evaluation

As machine translation (MT) systems—especially those powered by Large Language Models (LLMs)—approach human-level quality, traditional metrics like BLEU or even neural metrics like COMET are hitting a wall. They provide a single number but offer no explanation as to why one translation is better than another.

Furthermore, when we use LLMs as "judges" to rank translations, they suffer from position bias: they often reflexively prefer the first or last translation they see, regardless of quality. The authors aim to create an evaluation system that provides actionable, multidimensional feedback (based on the MQM framework) while remaining robust against these ordering artifacts.

Methodology: The TransEvalnia System

The core of the paper is the TransEvalnia system, which follows a structured reasoning path. The authors compare three primary approaches: a single-step method, a two-step method, and the proposed three-step interleaving strategy.

Comparison of evaluation methods: (a) single-step, (b) two-step, and (c) three-step interleaving.
  1. Span-based Evaluation: Instead of looking at a whole paragraph, the system breaks translations into smaller "spans" and evaluates them across six dimensions: Accuracy, Terminology, Linguistic Conventions, Audience Appropriateness, Hallucinations, and Missing Content.
  2. The Three-Step Interleaving Strategy: To fight position bias, the authors developed a sophisticated prompting pipeline:
  3. Step 1: Evaluate each translation independently.
  4. Step 2 (The Interleave): Take the evaluations from Step 1 and "braid" them together by category (e.g., show Accuracy for Translation A, then Accuracy for Translation B, then Terminology for A, then Terminology for B).
  5. Step 3: Ask the LLM to rank the translations based on this interleaved evidence.
  6. Scoring: The system also assigns 1–5 Likert scale scores for each dimension and an overall score.

The authors tested this using top-tier models like Claude-3.5-Sonnet and Qwen-2.5-72B-Instruct.

Datasets and Benchmarks

The study utilizes a mix of established benchmarks and challenging new sets:

  • Generic (en-ja): A new dataset combining English proverbs (hard to translate) and news sentences.
  • Haiku (ja-en): 100 haikus by Matsuo Bashō, requiring the evaluation of "Emotional Content" rather than just literal accuracy.
  • Hard en-ja: 10 extremely tricky sentences specifically designed to trip up MT systems, rated by human experts.
  • WMT Shared Tasks (2021–2024): Standard industry benchmarks covering language pairs like English-German, Chinese-English, and English-Spanish.

Key Insights

  • Reasoning as a Debugging Tool: By forcing the LLM to explain its logic, users can "evaluate the evaluation." If the LLM makes a mistake in its reasoning, a human can easily spot it, making the system more transparent than a raw score.
  • Interleaving is the "Secret Sauce": Simply asking an LLM to rank two translations (the "one-step" method) is highly susceptible to bias. By interleaving the evaluations, the "signal" for both translations is spread across the model's context window, forcing it to compare them side-by-side at every step.
  • LLMs vs. Humans: In some cases, the LLM-based evaluation ranked LLM-generated translations higher than the human reference translations. Human raters often agreed with this, confirming that modern MT can indeed surpass human references in specific contexts.

Key Results

TransEvalnia performed on par with or better than the current state-of-the-art MT-Ranker across most WMT datasets. Professional translators agreed with TransEvalnia’s fine-grained span evaluations roughly 85% of the time.

Performance comparison of TransEvalnia variants against state-of-the-art metrics across multiple datasets.

The three-step interleaved approach consistently yielded the lowest "bias inconsistency" scores compared to simpler one-step or two-step methods. Scores from Claude-3.5-Sonnet showed a Spearman correlation with human raters (~0.54) that was comparable to the correlation between two different human rater groups (~0.52).

Limitations and Future Work

While interleaving helps, it doesn't completely eliminate position bias. The authors suggest that transformer-specific issues like positional encodings and causal masking still play a role. Additionally, a three-step reasoning process is significantly more expensive and slower than a single-pass metric like BLEU or COMET. Future work will investigate how positional encoding spans in transformers contribute to bias and how to further refine open-source models for this specific task.

Summary updated

Abstract

We present TransEvalnia, a prompting-based translation evaluation and ranking system that uses reasoning in performing its evaluations and ranking. This system presents fine-grained evaluations based on a subset of the Multidimensional Quality Metrics (https://themqm.org/), returns an assessment of which translation it deems the best, and provides numerical scores for the various dimensions and for the overall translation. We show that TransEvalnia performs as well as or better than the state-of-the-art MT-Ranker (Moosa et al. 2024) on our own English-Japanese data as well as several language pairs from various WMT shared tasks. Using Anthropic's Claude-3.5-Sonnet and Qwen-2.5-72B-Instruct as the evaluation LLMs, we show that the evaluations returned are deemed highly acceptable to human raters, and that the scores assigned to the translations by Sonnet, as well as other LLMs, correlate well with scores assigned by the human raters. We also note the sensitivity of our system -- as well as MT-Ranker -- to the order in which the translations are presented, and we propose methods to address this position bias. All data, including the system's evaluation and reasoning, human assessments, as well as code is released.
BibTeX
@misc{sproat2025transevalnia,
  author = {Richard Sproat and Tianyu Zhao and Llion Jones},
  title = {{TransEvalnia: Reasoning-based Evaluation and Ranking of Translations}},
  year = {2025},
  eprint = {2507.12724},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.12724},
}
Figure