HUME: Human UCCA-Based Evaluation of Machine Translation

Alexandra Birch, Barry Haddow, Ondrej Bojar, Omri Abend

Introduction

Evaluating machine translation (MT) quality is a persistent challenge in natural language processing. Traditional human evaluation methods, such as Direct Assessment (0-100 scoring) or Relative Ranking, often suffer from two primary issues: the "Black Box" problem, where a single score fails to localize specific errors, and the "Scalability" problem, where human agreement drops significantly as sentence length increases.

To address these limitations, the authors propose HUME (Human UCCA-based MT Evaluation). HUME acts as a "semantic microscope," decomposing sentences into discrete semantic units to provide a fine-grained, reliable analysis of translation quality.

The UCCA Framework

At the core of HUME is UCCA (Universal Conceptual Cognitive Annotation), a semantic representation framework that breaks a sentence into a graph of "Scenes" (actions or states) and "Participants." Unlike previous metrics that focused strictly on verb-argument structures, UCCA provides broader coverage, including linkers, nouns, and adjectives, which are often critical to preserving meaning in specialized domains.

An example of UCCA semantic annotation for the sentence "After graduation Tom moved to America."

The HUME Evaluation Process

The HUME workflow consists of three distinct stages:

  1. Source Annotation: Experts first annotate the source sentence using UCCA. This creates a "gold standard" structure that is used for all subsequent translation evaluations.
  2. Bilingual Judgment: A bilingual annotator evaluates each semantic unit in the source against the MT output. Atomic units (words or phrases) are assigned "traffic light" ratings: Green (correct), Orange (partially correct), or Red (incorrect). Structural relationships are marked as Adequate or Bad.
  3. Aggregation: These individual judgments are mathematically combined into a single composite score that reflects the overall adequacy of the translation.
The HUME annotation interface allows bilingual judges to assign "traffic light" scores to specific semantic units.

By evaluating the MT output against the clean source structure rather than a potentially garbled translation, HUME avoids the difficulty of parsing poorly translated text directly.

Experimental Results and Insights

The researchers validated HUME in the medical domain, evaluating English source texts from NHS 24 and Cochrane reviews translated into Czech, German, Polish, and Romanian.

Key Findings:

  • Reliability and Stability: Unlike traditional ranking methods, HUME’s inter-annotator agreement (Cohen’s Kappa 0.6–0.7) remains stable even as sentences grow longer. This makes it particularly suitable for complex, professional documentation.
  • Validity: HUME scores show a strong Pearson correlation (up to 0.78) with traditional Direct Assessment scores, confirming that aggregating fine-grained semantic judgments accurately reflects overall translation quality.
  • Reference-Free Evaluation: Because HUME compares the MT output directly to the source, it eliminates "reference bias," where valid translations are penalized for using different synonyms than a single human-written reference.

Conclusion and Future Work

While HUME currently requires bilingual annotators and takes approximately 2 to 4 minutes per sentence, it provides actionable data that sentence-level scores cannot. Future work aims to automate the process through high-quality automatic UCCA parsing and to refine the aggregation formula by weighting "Centers" (core phrase components) more heavily than modifiers to better reflect communicative importance.

Summary updated

BibTeX
@inproceedings{birch2016hume,
  author = {Alexandra Birch and Barry Haddow and Ondrej Bojar and Omri Abend},
  title = {{HUME: Human UCCA-Based Evaluation of Machine Translation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2016},
  doi = {10.18653/v1/d16-1134},
}
Figure