Introduction
Evaluating machine translation (MT) quality is a persistent challenge in natural language processing. Traditional human evaluation methods, such as Direct Assessment (0-100 scoring) or Relative Ranking, often suffer from two primary issues: the "Black Box" problem, where a single score fails to localize specific errors, and the "Scalability" problem, where human agreement drops significantly as sentence length increases.
To address these limitations, the authors propose HUME (Human UCCA-based MT Evaluation). HUME acts as a "semantic microscope," decomposing sentences into discrete semantic units to provide a fine-grained, reliable analysis of translation quality.
The UCCA Framework
At the core of HUME is UCCA (Universal Conceptual Cognitive Annotation), a semantic representation framework that breaks a sentence into a graph of "Scenes" (actions or states) and "Participants." Unlike previous metrics that focused strictly on verb-argument structures, UCCA provides broader coverage, including linkers, nouns, and adjectives, which are often critical to preserving meaning in specialized domains.
The HUME Evaluation Process
The HUME workflow consists of three distinct stages:
- Source Annotation: Experts first annotate the source sentence using UCCA. This creates a "gold standard" structure that is used for all subsequent translation evaluations.
- Bilingual Judgment: A bilingual annotator evaluates each semantic unit in the source against the MT output. Atomic units (words or phrases) are assigned "traffic light" ratings: Green (correct), Orange (partially correct), or Red (incorrect). Structural relationships are marked as Adequate or Bad.
- Aggregation: These individual judgments are mathematically combined into a single composite score that reflects the overall adequacy of the translation.
By evaluating the MT output against the clean source structure rather than a potentially garbled translation, HUME avoids the difficulty of parsing poorly translated text directly.
Experimental Results and Insights
The researchers validated HUME in the medical domain, evaluating English source texts from NHS 24 and Cochrane reviews translated into Czech, German, Polish, and Romanian.
Key Findings:
- Reliability and Stability: Unlike traditional ranking methods, HUME’s inter-annotator agreement (Cohen’s Kappa 0.6–0.7) remains stable even as sentences grow longer. This makes it particularly suitable for complex, professional documentation.
- Validity: HUME scores show a strong Pearson correlation (up to 0.78) with traditional Direct Assessment scores, confirming that aggregating fine-grained semantic judgments accurately reflects overall translation quality.
- Reference-Free Evaluation: Because HUME compares the MT output directly to the source, it eliminates "reference bias," where valid translations are penalized for using different synonyms than a single human-written reference.
Conclusion and Future Work
While HUME currently requires bilingual annotators and takes approximately 2 to 4 minutes per sentence, it provides actionable data that sentence-level scores cannot. Future work aims to automate the process through high-quality automatic UCCA parsing and to refine the aggregation formula by weighting "Centers" (core phrase components) more heavily than modifiers to better reflect communicative importance.
Summary updated