TL;DR
Unbabel's COMET framework, a neural metric for evaluating machine translation quality, has been extended to handle document-level evaluation and multi-reference scenarios. Their participation in WMT20 demonstrated state-of-the-art performance across various tracks, highlighting the importance of source text and reference quality for robust MT evaluation.
What Problem Does This Paper Address?
Evaluating the quality of machine translation (MT) automatically is a long-standing challenge. Traditional metrics like BLEU, while useful, often fail to capture nuanced aspects of translation quality that humans perceive. This paper addresses the need for more accurate and reliable MT evaluation metrics, particularly in the context of the WMT (Workshop on Machine Translation) shared tasks. Specifically, it focuses on improving segment-level, document-level, and system-level evaluation, and also tackles the "Quality Estimation as a Metric" task. The authors aim to build upon existing neural approaches, like their own COMET framework, to achieve higher correlations with human judgments.
Methods
The core of Unbabel's contribution is the COMET framework, a neural metric that leverages pre-trained language models (specifically XLM-ROBERTa) to generate quality scores. For this work, they've extended COMET in several ways:
- Estimator Models: These models are trained to directly predict human quality scores (like Direct Assessment, HTER, and MQM) by regressing on segment-level representations. They've fine-tuned XLM-ROBERTa with larger hidden sizes and a layer-wise learning rate decay to improve generalization.
- Translation Ranking Model: This model is trained to rank MT outputs, rather than predict absolute scores. For this task, they opted to use the original COMET-RANK model without alterations, as larger encoders led to instability.
- Document-Level Score: A novel technique is proposed to derive a document-level score by taking a weighted average of segment-level predictions, weighted by segment length. This is computed by processing the entire document as a single batch for efficiency.
- Multi-Reference Handling: To leverage multiple references, they introduce a method where the model is trained to treat source and reference inputs interchangeably. At inference, they pass the source, hypothesis, and each reference through the model in various permutations and aggregate the predictions.
- Quality Estimation as a Metric: For this track, they adapted their COMET framework by removing the reference input and reducing the dimensionality of the feed-forward network.
Datasets
The models were trained and evaluated on various datasets:
- HTER Corpora: A concatenation of QT21 and APE-QUEST corpora, providing segments from IT, life sciences, and legal domains.
- MQM Corpus: An internal dataset of customer support chat messages with MQM annotations.
- DA Corpora: Human judgments from WMT News Translation shared tasks (2017, 2018, and 2019), including direct assessments and relative rankings.
- WMT19 Test Sets: Used for evaluating the models on segment-level, document-level, and system-level tasks.
- Additional References: For specific language pairs (en-de, de-en), they utilized extra references from Freitag et al. (2020) and the WMT19 News shared task test set.
Key Insights and Ideas
- Source Text is Crucial: Including the source text as input to the MT evaluation model significantly benefits its performance, a finding reinforced by their work.
- Reference Quality Matters: The quality of translation references is more important than quantity. Using lower-quality additional references can sometimes hurt performance compared to using a single high-quality reference.
- COMET's Versatility: The COMET framework, initially designed for segment-level evaluation, can effectively be adapted for document-level metrics and quality estimation tasks.
- Ranking vs. Regression: While larger models improve regression-based metrics (Estimators), they can lead to instability in ranking models.
- System-Level Robustness: A pairwise comparison approach for system-level evaluation, using Kendall's Tau, proves more robust to outliers and better reflects real-world scenarios than simple Pearson correlation.
Key Results
Unbabel's COMET models achieved state-of-the-art or highly competitive results across various WMT20 Metrics Shared Task tracks.
- Segment-Level: COMET models consistently outperformed prior metrics on DARR data for both English-to-other and other-to-English language pairs, often by significant margins.
- System-Level: COMET models achieved strong Pearson correlation scores with human judgments, outperforming most other metrics.
- Document-Level: The proposed micro-averaging technique for document-level scores showed superior performance compared to simple unweighted averages.
- Multi-Reference Handling: Experiments with additional references showed promising results, with the DA Estimator model benefiting from higher-quality references.
Limitations and Future Work
- COMET-RANK (base) Performance: The COMET-RANK (base) model did not perform as well at the system-level as expected, which the authors attribute to training directly on DARR data. They suggest this might be due to the ambiguity in the DARR scale for less distinct differences between hypotheses.
- Multi-Reference Data: While experiments with multi-reference handling were promising, the authors acknowledge a lack of statistically significant conclusions due to limited data.
- Future Work: Further investigation into how to better leverage different kinds of references is suggested as an interesting direction for future research.
Summary updated