Tagged Span Annotation for Detecting Translation Errors in Reasoning LLMs

Taemin Yeom, Young‐Hee Ryu, Yoonjung Choi, JinYeong Bak2025 · DOI

TL;DR

Researchers have developed Tagged Span Annotation (TSA), a new prompting strategy that helps reasoning LLMs like OpenAI’s o3 pinpoint specific errors in machine translations. By using numbered inline tags (e.g., <v0>wrong word</v0>) instead of character offsets or raw text, their reference-free system achieved a character-level F1 score of 27.05—crushing the previous reference-based state-of-the-art (XCOMET-XXL) which scored around 17.

What Problem Does This Paper Address?

Evaluating machine translation (MT) has moved beyond simple "quality scores" toward fine-grained error detection. We want to know exactly where a translation failed (the span), why it failed (the category), and how bad it is (the severity).

Current Large Language Models (LLMs) used as "judges" struggle with this for two reasons:

  1. Ambiguity: If an LLM just lists the word "bank" as an error, but "bank" appears three times in the sentence, we don't know which one is wrong.
  2. Mapping Issues: LLMs are notoriously bad at providing exact character indices (e.g., "error starts at character 12") because they process text as tokens, not raw characters.

Methods

The authors introduce Tagged Span Annotation (TSA). Instead of asking the model for coordinates, they ask the model to rewrite the translation with numbered inline tags surrounding the errors.

  • The TSA Format: A translation might look like: He arrived on <v0>Monday</v0> but forgot <v1></v1> the documents.
  • The Metadata: The model then provides a structured JSON object mapping those tags to details:
  • 0: {severity: major, category: mistranslation}
    • 1: {severity: minor, category: omission} (Note the empty tag <v1></v1> used to mark where a word was forgotten).
  • Precision Emphasis: To fix a common issue where LLMs "over-flag" errors (high recall, low precision), the authors added specific instructions to the system prompt: "Only mark errors when you are confident" and "Mark only the minimal substring."
  • Reasoning Models: The system was tested on OpenAI’s o3, o4-mini, and GPT-4.1.

Datasets

The study used a refined version of the WMT24 ESA (Error Span Annotation) dataset.

  • Scope: Six language pairs (English to Czech, Japanese, Chinese, Icelandic, Ukrainian, and Russian).
  • Size: Approximately 39,685 segments.
  • Refinement: The authors cleaned the human gold-standard annotations to ensure they were suitable for span-level evaluation.

Key Insights and Ideas

  • The "Reasoning" Advantage: The most striking finding was that TSA only provides a massive boost to reasoning models (like o3). For standard models (like GPT-4.1), TSA actually performed slightly worse than simpler formats. The authors hypothesize that for reasoning models, the act of "tagging" the text serves as an externalized intermediate reasoning step, helping the model "think" through the error before classifying it.
  • Reference-Free > Reference-Based: Traditionally, the best evaluation tools are "reference-based" (they compare the machine output to a human-written "perfect" translation). This system is reference-free (it only looks at the source and the machine output) yet it significantly outperformed the best reference-based encoder models.
  • Minimal Spans Matter: LLMs tend to highlight entire phrases as errors. By forcing the model to select the "minimal substring," the researchers significantly improved the character-level F1 score.

Key Results

  • Top Performance: The o3 model using TSA achieved an average F1 score of 27.05.
  • Comparison: This outperformed XCOMET-XXL (F1 17.30) and a "fixed" version of the GEMBA baseline (F1 16.50).
  • Ablation Success: Using TSA (inline tags) was superior to "Direct-index" (asking for character numbers) by nearly 5 F1 points, proving that the format of the output is just as important as the model's "intelligence."

Limitations and Future Work

  • Cost: Using high-end reasoning models is expensive. The authors estimated it cost about $139 to evaluate ~40k segments. This makes it a great tool for creating "gold standard" data, but perhaps too pricey for real-time, high-volume translation monitoring.
  • Proprietary Models: The research focused on OpenAI models. The authors acknowledge the need to test these prompting strategies on open-source models (like Llama or DeepSeek) to improve reproducibility.
  • Low-Resource Languages: While the system performed well across the board, the causes of performance variance in lower-resource languages remain a topic for future investigation.
  • Integration: The authors suggest that this TSA system could be plugged into "Automatic Post-Editing" (APE) systems to not only detect errors but automatically fix them.

Summary updated

Abstract

We present the submission of the AIP team to the WMT 2025 Unified MT Evaluation Shared Task, focusing on the span-level error detection subtask.Our system emphasizes responseformat design to better harness the capabilities of OpenAI's o3, the state-of-the-art reasoning LLM.To this end, we introduce Tagged Span Annotation (TSA), an annotation scheme designed to more accurately extract span-level information from the LLM.On our refined version of WMT24 ESA dataset, our referencefree method achieves an F1 score of approximately 27 for character-level label prediction, outperforming the reference-based XCOMET-XXL at approximately 17.
BibTeX
@misc{yeom2025tagged,
  author = {Taemin Yeom and Young‐Hee Ryu and Yoonjung Choi and JinYeong Bak},
  title = {{Tagged Span Annotation for Detecting Translation Errors in Reasoning LLMs}},
  year = {2025},
  doi = {10.18653/v1/2025.wmt-1.62},
}
Figure