UniTE: Unified Translation Evaluation

Yu Wan, Dayiheng Liu, Baosong Yang, Haibo Zhang, Boxing Chen, Derek F. Wong, Lidia S. Chao

Abstract

Machine translation (MT) evaluation has traditionally relied on specialized metrics tailored to specific input scenarios: reference-only, source-only (Quality Estimation), and combined source-reference evaluation. This paper introduces UniTE (Unified Translation Evaluation), the first framework designed to master all three evaluation tasks within a single model. By leveraging a novel Monotonic Regional Attention (MRA) mechanism and a ranking-based pretraining strategy on synthetic data, UniTE consistently outperforms specialized state-of-the-art metrics across major WMT benchmarks. Our findings suggest that a unified approach can effectively utilize cross-task knowledge transfer to achieve superior robustness and accuracy.

Introduction

Evaluating the quality of machine translation is a fragmented field. Depending on the availability of a reference translation, a source sentence, or both, researchers typically employ different tools. This fragmentation is not only inconvenient but also overlooks the underlying commonalities between these evaluation tasks. UniTE addresses this by providing a high-performance, unified framework built upon a multilingual Pretrained Language Model (PLM), specifically XLM-R.

The Three Pillars of MT Evaluation

UniTE is designed to handle three distinct scenarios based on the available input segments:

  1. Reference-only (REF): Comparing the model's hypothesis ($h$) against a gold-standard reference ($r$).
  2. Source-only (SRC): Estimating quality based only on the source sentence ($s$), a task known as Quality Estimation (QE).
  3. Combined (SRC+REF): Utilizing both the source and reference segments to judge the hypothesis.

Methodology

Unified Input Architecture

UniTE simplifies the evaluation process by concatenating all available segments ($h, s, r$) into a single sequence. To maintain structural consistency across tasks, the hypothesis is always positioned at the beginning of the input.

Monotonic Regional Attention (MRA)

While standard PLMs utilize full attention, allowing every token to interact with every other token, the authors discovered that unrestricted interaction between the source and hypothesis can introduce noise in combined tasks. To mitigate this, UniTE introduces Monotonic Regional Attention (MRA). This mechanism uses a mask to restrict "semantic flows," forcing the model to focus on the most relevant cross-segment alignments (e.g., $h \rightarrow r$, $s \rightarrow r$, and $h \rightarrow s$) while preventing bidirectional noise.

The UniTE architecture and its application across the three evaluation tasks.

Unified Pretraining and Ranking-based Labeling

To bridge the gap between general language modeling and translation evaluation, UniTE employs a two-stage training process. First, it uses synthetic data generated by "downgrading" high-quality translations through word and span dropping. Second, instead of relying on potentially biased raw scores from other metrics, the authors implement a Borda count-based ranking strategy to label this synthetic data. This approach normalizes distributions across different languages and systems, providing a cleaner training signal for the model.

Experimental Results

UniTE was evaluated on the WMT 2019 Metrics Shared Task and the WMT 2020 Quality Estimation Task. The results demonstrate clear dominance over specialized baselines.

Performance on English-Targeted Tasks

In the WMT 2019 Metrics task, the UniTE-UP variant (utilizing unified pretraining) outperformed specialized baselines such as RoBERTa+Concat and COMET across all three evaluation settings. The model showed a significant improvement in Kendall’s Tau correlation, particularly in the combined evaluation scenario.

Kendall’s Tau correlation results on English-targeted language pairs.

Multilingual and Zero-Shot Capabilities

The multilingual version, UniTE-MUP, achieved new state-of-the-art results on multilingual pairs and demonstrated strong zero-shot capabilities. In the WMT 2020 QE task, UniTE-MUP achieved a Pearson correlation of 67.8%, surpassing the previous winner, TransQuest (67.3%), while maintaining its unified architecture.

Discussion and Limitations

A key insight from this research is the synergy achieved through unification; training on all three tasks simultaneously improves performance on each individual task. However, the authors noted a "curse of multilingualism," where the multilingual model slightly underperforms the English-specific version on English-only tasks due to vocabulary dilution. Additionally, while MRA is highly effective, its relative benefit may decrease if massive amounts of high-quality human-labeled data become available for the combined task.

Conclusion

UniTE represents a significant step toward a more integrated approach to machine translation evaluation. By unifying disparate tasks into a single framework and introducing controlled interaction through MRA, it provides a robust and versatile tool that surpasses the performance of specialized state-of-the-art metrics.

Summary updated

BibTeX
@inproceedings{wan2022unite,
  author = {Yu Wan and Dayiheng Liu and Baosong Yang and Haibo Zhang and Boxing Chen and Derek F. Wong and Lidia S. Chao},
  title = {{UniTE: Unified Translation Evaluation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.558},
}

Loading related content...

Figure