TL;DR
The WMT19 Metrics Shared Task evaluated automatic machine translation (MT) evaluation metrics, revealing that while many metrics correlate well with human judgments on broad system sets, their effectiveness significantly degrades when evaluating top-performing MT systems. This highlights a critical challenge: current metrics struggle to differentiate between high-quality translations, suggesting a need for metrics that better capture nuanced aspects of translation quality.
What Problem Does This Paper Address?
Evaluating the quality of machine translation (MT) is crucial for developing better MT systems. While automatic metrics like BLEU are widely used for their speed and cost-effectiveness compared to human evaluation, their reliability, especially when assessing the outputs of cutting-edge MT systems, has been a long-standing question. The WMT Metrics Shared Task aims to address this by rigorously evaluating a wide range of automatic metrics against human judgments, specifically investigating how their performance changes when faced with a diverse set of MT systems versus a more homogeneous set of top-tier systems. This is important because the ultimate goal is to have metrics that can reliably guide the development of the best possible MT systems, not just distinguish between very poor ones.
Methods
The WMT19 Metrics Shared Task involved two main evaluation levels: system-level and segment-level.
- System-Level Evaluation: Metrics were assessed based on their correlation with human judgments of overall system performance. The primary metric used was Pearson correlation, with statistical significance tested using Williams' test.
- Segment-Level Evaluation: Metrics were evaluated on their ability to predict human judgments of individual sentence translations. Due to limitations in collecting sufficient direct assessments (DA) for every segment, the task employed a "DA relative ranking" (daRR) approach, converting DA scores into pairwise preferences. Kendall's Tau-like formulation was used to measure metric correlation with daRR judgments.
A key aspect of this year's task was the inclusion of "QE as a Metric" submissions, where quality estimation systems were evaluated using the same metrics framework but without access to reference translations.
Datasets
The evaluation used the newstest2019 dataset, which is part of the WMT19 News Translation Task. This dataset comprises source texts, MT system outputs, and reference translations. For the "QE as a Metric" track, reference translations were withheld. The dataset covered 18 language pairs, with varying numbers of sentences per pair.
Key Insights and Ideas
- Top Systems Pose a Challenge: The most striking finding is that many metrics, including widely used ones like BLEU and SacreBLEU, show a significant drop in correlation with human judgments when evaluating only the top-performing MT systems. This suggests that current metrics are better at distinguishing between systems of widely varying quality than between systems that are already very good.
- Reference-Based vs. Reference-Free Metrics: The task evaluated both reference-based metrics (which compare MT output to human references) and reference-less metrics (often from the Quality Estimation track), assessing their performance in different scenarios.
- Neural Metrics Show Promise: Metrics leveraging neural networks and contextual word embeddings (like ESIM and YISI variants) generally performed better, especially when evaluating top systems, indicating that semantic understanding is becoming increasingly important for effective MT evaluation.
- Implementation Matters: Differences in metric implementations (e.g., BLEU and CHRF) led to variations in scores and correlations, highlighting the need for standardized implementations.
Key Results
- System-Level Correlations: For many language pairs, the best metrics achieved Pearson correlations above 0.95 when considering all systems. However, this correlation often dropped significantly when only the top-N systems were analyzed. For example, SacreBLEU-BLEU's correlation with human judgment for English-German dropped from 0.969 with all systems to below 0.5 when considering only the top 10 systems, and even became negative for the top 4 systems.
- Segment-Level Correlations: Segment-level metrics showed more varied performance. YISI-1 SRL was identified as a "winner" for many language pairs, and ESIM also performed strongly. QE systems, while generally showing lower correlations than standard metrics, sometimes exhibited upward trends with top-N systems, suggesting they might be better suited for evaluating high-quality outputs.
- Metric Wins: The YISI metrics series achieved the highest correlations in several language pairs and were not significantly outperformed by any other metric. ESIM performed best on 5 language languages, obtaining 11 "wins" out of 16 language pairs. EED also showed strong performance for out-of-English language pairs.
Limitations and Future Work
- Sensitivity to System Sets: The study emphasizes that metric correlations are heavily influenced by the set of MT systems being evaluated. The observed correlations can be overly optimistic when a wide range of system qualities is included.
- Need for Better Metrics: The degradation of metric performance on top systems points to a clear need for new metrics that can better capture subtle differences in translation quality, potentially by incorporating deeper semantic understanding.
- Future Research Directions: The authors suggest future work should explore whether semantic features, particularly contextual word embeddings, are key to achieving better metric performance. They also call for standardized metric implementations to ensure long-term comparability and reproducibility. The investigation into why BLEU performs well in some contexts but not others is also highlighted as an area for future research.
Summary updated