TL;DR
This paper presents the results of the WMT18 Metrics Shared Task, where researchers evaluated various automatic machine translation (MT) metrics against human judgments. The study found that both character-level and embedding-based metrics achieved strong correlations with human evaluations, highlighting the evolving landscape of MT evaluation.
What Problem Does This Paper Address?
Accurate evaluation of machine translation (MT) systems is crucial for tracking progress and developing better models. While human evaluation is the gold standard, it's expensive and time-consuming. Automatic metrics aim to provide a faster, cheaper alternative by comparing MT outputs to human-produced reference translations. However, the effectiveness of these automatic metrics varies, and it's essential to regularly assess their performance against human judgments to identify the most reliable ones. The WMT Metrics Shared Task serves this purpose by bringing together researchers to compare their metrics on a standardized dataset and evaluation framework.
Methods
The WMT18 Metrics Shared Task employed two main evaluation approaches:
- System-Level Evaluation: Metrics were assessed on their ability to predict the overall quality of an MT system's output for an entire test set. This was done by calculating the Pearson correlation between the metric's scores and human judgments (Direct Assessment - DA).
- Segment-Level Evaluation: Metrics were evaluated on their ability to predict the quality of individual sentences. Due to insufficient data for direct comparison, a "relative ranking" (DARR) approach was used, where DA scores were converted into pairwise preferences. Kendall's Tau-like formulation was then used to measure the correlation between metric predictions and these human preferences.
A key methodological aspect was the use of Direct Assessment (DA) as the "golden truth" for human judgments. DA simplifies the task by asking annotators to assess the quality of a single MT output in isolation, relative to a reference translation, on a continuous scale. This contrasts with older methods like "relative ranking" (RR) where annotators compared multiple system outputs simultaneously.
To increase the robustness of system-level evaluations, especially for language pairs with few participating MT systems, the task also utilized hybrid systems. These were created by randomly combining outputs from two existing MT systems, generating a larger set of systems to evaluate against.
Datasets
The primary dataset used was newstest2018, which was also used in the WMT18 News Translation Task. This dataset consisted of approximately 3,000 sentences for each language pair (with some exceptions for Chinese and Estonian), each with a single reference translation.
Human judgments were collected through Direct Assessment (DA) from MT researchers, in-house evaluators, and crowd-sourced workers on Amazon Mechanical Turk. These judgments were then processed to create the "golden truth" for both system-level and segment-level evaluations.
Key Insights and Ideas
- Character vs. Embedding-based Metrics: The study found that both character-level metrics (like BEER, CHARACTER) and embedding-based metrics (like RUSE, YISI) performed well, indicating a shift towards more sophisticated linguistic feature extraction.
- Hybrid Systems Enhance Robustness: The use of hybrid systems proved effective in strengthening system-level evaluations, particularly for language pairs with a limited number of participating MT systems. This allowed for more reliable comparisons between metrics.
- DA as the New Standard: The continued reliance on Direct Assessment (DA) for human judgments signifies its acceptance as a robust and more reliable method for evaluating MT quality compared to older techniques.
- Segment-Level Challenges: The task faced challenges in collecting sufficient segment-level human judgments, necessitating the conversion of DA scores to a relative ranking format (DARR). This highlights the ongoing difficulty in obtaining comprehensive segment-level human evaluation data.
- Metric Performance Varies by Language Pair: Metric performance was not uniform across all language pairs, underscoring the importance of language-specific evaluations and the need for metrics that generalize well.
Key Results
- System-Level Performance: Several metrics achieved high Pearson correlations (above 0.9) with human judgments on system-level evaluations, with some reaching over 0.95 for specific language pairs. Metrics like RUSE and YISI stood out for their consistent performance across multiple language pairs.
- Segment-Level Performance: Segment-level correlations (measured by Kendall's Tau) were generally lower, typically ranging from 0.25 to 0.4. The best-performing metrics at this level could reach up to 0.69 correlation.
- Notable Metrics: RUSE and YISI were highlighted as strong performers, particularly for their consistent results across various language pairs. ITER also showed good performance but had some language-specific weaknesses.
- Availability: A positive outcome was that many of the top-performing metrics were publicly available, encouraging their wider adoption and use.
Limitations and Future Work
- Limited Sample Size for System-Level: For some language pairs, the number of participating MT systems was small (as few as five), which could lead to inconclusive results and make it difficult to identify statistically significant differences between metrics.
- Segment-Level Data Constraints: The task was unable to fully implement the standard segment-level DA evaluation methodology due to insufficient human judgments. The reliance on converted DA scores (DARR) for segment-level evaluation is noted as a limitation. The authors express a desire to return to the standard DA evaluation in the future, requiring a minimum of 15 judgments per segment.
- Hybrid System Limitations: While effective for system-level evaluation, the current method of creating hybrid systems by randomly mixing sentences might not be suitable for document-level evaluation, as it could break cross-sentence references.
- Future Work: The authors encourage metric developers to contribute their code to the Moses scorer to ensure their long-term inclusion in future WMT tasks. They also suggest that future evaluations might consider document-level assessment.
Summary updated