Unbabel's Participation in the WMT20 Metrics Shared Task

Ricardo Rei, Craig Stewart, Ana C. Farinha, Alon Lavie

TL;DR

Unbabel's COMET framework, a neural metric for machine translation evaluation, achieved state-of-the-art results across multiple tracks at the WMT20 Metrics Shared Task. The team introduced novel methods for handling multiple references and calculating document-level scores, demonstrating COMET's robustness and effectiveness compared to existing metrics.

What Problem Does This Paper Address?

Evaluating the quality of machine translation (MT) is a notoriously difficult task. Traditional metrics like BLEU, while widely used, often fail to capture nuanced aspects of translation quality and can be easily gamed. The WMT Metrics Shared Task aims to push the boundaries of MT evaluation by developing metrics that better correlate with human judgments. This paper addresses the challenge of creating automatic MT evaluation metrics that are not only accurate at the segment level but also reliable for document-level and system-level assessments, and can effectively leverage modern pre-trained language models.

Methods

The core of Unbabel's contribution is their COMET framework, which builds upon large pre-trained language models like XLM-ROBERTa. The framework offers two main types of models:

  • Estimator Models: These models are trained to directly predict human-assigned quality scores (like Direct Assessment, HTER, or MQM) for MT outputs. For this work, they used XLM-ROBERTa large and increased the feed-forward hidden sizes, while employing a layer-wise learning rate decay to improve generalization to new language pairs.
  • Translation Ranking Model: This model is trained to rank MT outputs or systems. Interestingly, they found that larger pre-trained encoders led to training instability for this specific task, so they opted for the previously proposed COMET-RANK model without alterations.

Beyond these core models, Unbabel introduced two key methodological innovations:

  1. Multi-Reference Handling: For language pairs with multiple human references, they developed a technique to leverage information from all available references at inference time. This involves creating multiple permutations of input embeddings and aggregating predictions, with an intuition that penalizing lower confidence might better align with human judgment.
  2. Document-Level Scoring: They proposed a simple yet effective method for calculating a document-level score by taking a weighted average of segment-level scores. This approach was found to reduce inference time by processing entire documents as a single batch.

Datasets

The paper details the use of several corpora for training and evaluation:

  • HTER Corpora: A concatenation of the QT21 corpus (IT and life sciences domains) and the APE-QUEST corpus (legal domain), providing ~211K tuples for training.
  • MQM Corpus: An extension of a proprietary corpus containing customer support chat messages, MQM-annotated, with ~27K tuples across 15 languages.
  • DA Corpora: Human judgments (Direct Assessments) collected annually by WMT since 2008. For recent years (2017-2019), these scores were mapped to relative rankings (DARR) due to a lack of annotators. Data from 2017 and 2018 was used for training, and 2019 for validation.
  • WMT19 Metrics Shared Task Test Sets: Used for evaluating the performance of their models.

Key Insights and Ideas

  • Pre-trained Models are Key: The success of COMET hinges on leveraging powerful pre-trained language models like XLM-ROBERTa, demonstrating their effectiveness beyond traditional NLP tasks.
  • Source Text Matters: Including the source text as input to the MT evaluation model was shown to be beneficial, a finding consistent with prior work.
  • Quality Over Quantity for References: For multi-reference handling, the quality of the additional references appears to be more critical than simply having more of them. Lower-quality references might even degrade performance.
  • Document-Level Aggregation Works: A simple weighted average of segment-level scores effectively captures document-level quality, offering a practical solution.
  • Ranker vs. Estimator Performance: An unexpected finding was that the COMET-RANK model didn't perform as well at the system level as the Estimator models, potentially due to the nature of DARR data and how it defines ambiguity thresholds.

Key Results

Unbabel's COMET models achieved state-of-the-art or highly competitive results across various tracks of the WMT20 Metrics Shared Task.

  • Segment-Level: COMET models consistently outperformed existing metrics like BLEU, CHRF, BERTSCORE, and PRISM on WMT19 test sets, often by significant margins, for both English-to-other and other-to-English language pairs.
  • System-Level: COMET models demonstrated strong Pearson correlation scores, outperforming most other metrics. Notably, the DA Estimator (large) model showed significant robustness in differentiating high-performing systems.
  • Document-Level: The document-level scores derived from COMET's segment-level estimators showed strong correlations with human judgments, with the HTER Estimator performing particularly well.
  • Multi-Reference Handling: While not showing significant gains in Kendall's Tau, the multi-reference technique was explored, with findings suggesting reference quality is paramount.

Limitations and Future Work

  • Ranker Instability: The COMET-RANK model exhibited training instability with larger pre-trained encoders, leading to a decision to use the previous version. This suggests further research is needed to optimize ranking models with these large architectures.
  • DARR Data Impact: The performance discrepancy between COMET-RANK's segment-level strength and system-level performance was attributed to the nature of DARR data, highlighting a potential limitation in how rankers learn from such data.
  • Reference Quality: The paper suggests that while leveraging multiple references is promising, the quality of these references is a critical factor, and further investigation into how to best utilize different kinds of references is an interesting direction for future work. The authors also note that their approach, designed for meaning representation, might inherently benefit less from additional references compared to lexical-based metrics.

Summary updated

BibTeX
@inproceedings{rei2020unbabel,
  author = {Ricardo Rei and Craig Stewart and Ana C. Farinha and Alon Lavie},
  title = {{Unbabel's Participation in the WMT20 Metrics Shared Task}},
  booktitle = {WMT @ EMNLP},
  year = {2020},
  url = {https://openreview.net/forum?id=1xGwgkhIC4A},
}

Loading related content...

Figure