Abstract
Current fine-grained error analyses by Large Language Models (LLMs) are gaining attention in machine translation, but these analyses often fail to ground errors in the specific reasons why annotated text spans are erroneous. If LLMs cannot provide these reasons, their corrections remain untrustworthy. This paper introduces a novel evaluation resource for the translation error grounding task, developed through a bi-directional grounding scheme. In the forward direction, we annotate explanations for error spans; in the backward direction, we identify error spans based on masked explanations. Consistency between these directions ensures valid, objective explanations. Our evaluation reveals that LLM grounding ability significantly lags behind human performance. Furthermore, we demonstrate that error grounding can effectively filter false alarms, significantly improving translation error detection.
Introduction
In the field of Machine Translation (MT), evaluation has evolved toward fine-grained analysis using frameworks like Multidimensional Quality Metrics (MQM). While MQM identifies where an error occurs and its type, it typically lacks the underlying linguistic reasoning—the "why." As LLMs are increasingly deployed to evaluate and refine translations, the lack of grounding creates a critical trust gap. If a model identifies an error but cannot justify it with linguistic facts, its subsequent corrections may be superficial or based on hallucinated reasoning. Currently, there is no benchmark to measure an LLM's ability to ground its error detection in actual linguistic facts.
The Bi-directional Grounding Scheme (BGS)
To address this, the researchers developed a Bi-directional Grounding Scheme (BGS). This framework regulates the creation of error explanations to minimize subjective bias and ensure mutual consistency between the error span and its justification.
The process consists of three main steps:
- Forward Grounding: Annotators write a detailed explanation for a known MQM error span, identifying the source span, target span, and the specific reason for the error.
- Backward Grounding: A different annotator (or an LLM) is given the source, the translation, and the explanation with the error span masked. They must then identify the original error span based solely on that explanation.
- Calibration: If the backward grounding does not point back to the original error, the explanation is deemed invalid and refined until consistency is achieved.
Standardizing Translation Error Explanations
To ensure high data quality and consistency, explanations were standardized using "basic elements" such as source spans (<s>), target spans (<t>), and the erroneous span (<e>). This structured approach allows for a rigorous "reasoning layer" on top of standard MQM annotations.
The researchers applied this scheme to existing MQM data from WMT 2022/2023 quality estimation tasks, focusing on Chinese-to-English (ZH-EN) and English-to-German (EN-DE) pairs. The final dataset contains approximately 2,000 manual grounding instances per language pair, providing a unique resource for testing model reasoning.
Key Results and Insights
The evaluation highlights a significant "grounding gap" between humans and state-of-the-art LLMs. While humans are excellent at "backward grounding" (finding the error from the reason), achieving nearly 90% accuracy, LLMs struggle significantly.
- LLM Performance: Even top-tier models like GPT-4 fail the backward grounding task nearly half the time, achieving only a 57.41% perfect match rate for ZH-EN. Smaller models like Llama 3.1-8B trail further behind.
- Succinctness vs. Verbosity: Manual explanations are more effective for grounding than "free-style" auto-generated ones. LLM-generated explanations tend to be over-long and include irrelevant information that distracts the model during the grounding process.
- Grounding as a Filter: The authors propose Iterative BGS as a way to improve automated error detectors like xCOMET. By checking if a model can consistently ground a detected error through an explanation loop, they can filter out "false alarms"—errors the model flags but cannot actually justify. This approach led to a significant boost in F1-scores, particularly in the partial set where GPT-4 improved the score from 54.5% to 55.6%.
Conclusion and Future Work
This research provides the first manual benchmark for grounding translation errors. The results underscore that while LLMs are becoming proficient at spotting errors, their ability to reason about them is still far below human levels. Future work should extend this framework to more diverse language pairs and specialized domains like legal or biomedical translation. Additionally, while the BGS framework mitigates subjectivity, linguistic explanations still carry inherent nuances that require ongoing refinement.
Summary updated