TL;DR
This paper argues that the standard way of evaluating machine translation (MT) systems using corpus-level aggregation of lexical metrics (like BLEU) is statistically flawed and correlates poorly with human judgments. The authors propose and demonstrate that segment-level aggregation offers a more robust and reliable evaluation, bridging the gap between traditional lexical metrics and newer neural metrics.
What Problem Does This Paper Address?
Evaluating machine translation (MT) systems is crucial for progress. Traditionally, metrics like BLEU have relied on aggregating scores across an entire test set (corpus-level aggregation, CLA). However, recent advancements have introduced neural metrics (like COMET, BLEURT) that often perform better and correlate more strongly with human assessments. This raises a question: are lexical metrics inherently inferior, or is there a flaw in how they are applied? This paper tackles the hypothesis that the aggregation method itself is the culprit. Specifically, it addresses the under-explored issue of how aggregating lexical metric scores (e.g., BLEU, chrF) at the corpus level versus the segment level impacts their correlation with human judgments and their statistical robustness. This is particularly important for low-resource languages where neural metrics are not yet widely applicable.
Methods
The core of the paper's methodology involves comparing two primary aggregation methods for lexical metrics:
- Corpus-Level Aggregation (CLA): This is the traditional approach where n-gram matches are computed across the entire test set, and a single global score is derived. The paper uses the
corpus_bleuandcorpus_chrffunctions from SacreBLEU for this. - Segment-Level Aggregation (SLA): This method involves calculating the metric score for each individual sentence (segment) and then averaging these segment scores to get the system-level score. The paper uses
sentence_bleuandsentence_chrffrom SacreBLEU.
To establish a benchmark for statistical robustness, the authors also employ Bootstrap Resampling (BRS), a computationally more expensive method that involves repeatedly resampling the test set and computing CLA scores to estimate score stability.
The authors then conduct empirical evaluations by:
- Computing correlations between CLA, SLA, and BRS scores for both BLEU and chrF.
- Analyzing the impact of test set size on the correlation between CLA and SLA scores, and their correlation with BRS.
- Comparing the correlations of these metrics (both lexical and neural) with human judgments (MQM and DA scores).
Datasets
The primary dataset used for evaluation is the WMT 2023 Metrics Shared Task dataset. This dataset is notable for its scale, containing 468,850 system outputs from 492 different MT systems across 147 language pairs and 48 source languages. The authors also utilize human annotations from the same WMT23 dataset, specifically Multidimensional Quality Measurements (MQM) and Direct Assessment (DA) scores, to serve as ground truth for evaluating metric performance.
Key Insights and Ideas
- CLA vs. SLA: A Fundamental Mathematical Difference: The paper highlights that CLA is akin to a "ratio of averages," while SLA is an "average of ratios." This mathematical distinction leads CLA to be biased by sentence length and the proportion of matches, whereas SLA treats each sentence's performance more equally.
- SLA is More Robust and Correlated to Humans: The most significant finding is that segment-level aggregation (SLA) consistently shows much stronger correlations with human judgments and more robust neural metrics (like BERTScore) compared to corpus-level aggregation (CLA).
- CLA Lacks Statistical Robustness: The experiments reveal that CLA scores are highly sensitive to the size of the test set, becoming statistically unreliable even with moderately sized test sets. In contrast, SLA scores remain robust and correlate well with BRS as the test set size increases.
- Lexical Metrics Aren't Inherently Flawed, Just Aggregated Poorly: The paper argues that the poor reputation of lexical metrics might be undeserved, stemming from the widespread but flawed adoption of CLA. SLA offers a way to revitalize their utility, especially for low-resource languages.
- SLA is a Cost-Effective Alternative: SLA provides statistically robust scores comparable to BRS but at a significantly lower computational cost.
Key Results
- Correlation with Human Judgments: SLA-based metrics (m-BLEU, m-chrF) significantly outperform their CLA counterparts (BLEU, chrF) in correlation with human judgments (MQM and DA scores). For instance, m-chrF achieved a correlation of 0.806 with MQM, a substantial improvement over chrF's 0.392.
- Statistical Robustness: When comparing CLA and SLA against bootstrap resampling (BRS), SLA consistently showed higher correlations as the test set size increased. CLA's correlation with BRS dropped significantly with larger test sets, indicating a lack of statistical robustness.
- Cross-Metric Correlations: SLA-based metrics (m-BLEU, m-chrF) showed strong correlations with each other (0.93), and also with their BRS counterparts (x-BLEU, x-chrF) (0.88 and 0.99 respectively). CLA-based metrics (BLEU, chrF) showed weaker correlations with these more robust methods.
Limitations and Future Work
- Single Data Source: The study relies on the WMT23 Metrics Shared Task dataset. While large and representative, the authors acknowledge that results might vary with different datasets.
- Single Tool: The experiments exclusively used SacreBLEU for BLEU and chrF calculations. While SacreBLEU is a de facto standard, other implementations might yield slightly different results.
- Default Parameters: The evaluation used default parameters for the metrics. Future work could explore how changing parameters (e.g., n-gram lengths) might interact with aggregation methods.
The authors strongly recommend that MT researchers abandon corpus-level aggregation in favor of segment-level aggregation for more reliable and human-aligned evaluations. They also suggest that SLA can serve as a computationally cheaper alternative to bootstrap resampling for statistical evaluation.
Summary updated