TL;DR
This paper introduces EED, a novel machine translation evaluation metric that extends the Levenshtein edit distance. EED aims to improve human correlation and computational efficiency by incorporating "jumps" and a coverage penalty, offering a faster and more reliable alternative to existing metrics like BLEU and TER.
What Problem Does This Paper Address?
Evaluating the quality of machine translation (MT) is crucial for developing better systems. While human evaluation is the gold standard, it's time-consuming and expensive. Consequently, automated metrics are essential for rapid development. Existing popular metrics like BLEU and TER have been around for decades, but with the rapid advancements in MT, there's a growing need for metrics that better correlate with human judgment and remain computationally efficient. The challenge lies in creating a metric that captures the nuances of translation quality without requiring extensive linguistic analysis or being overly slow to compute.
Methods
The paper proposes EED (Extended Edit Distance), which builds upon the Levenshtein edit distance. The core idea is to extend the standard edit operations (insertion, deletion, substitution) with two key novelties:
- Jumps: Inspired by metrics like CDER, EED allows for "jumps" between characters or blocks of characters. Crucially, in EED, these jumps are restricted to occur only when a blank character is encountered in the reference sentence. This allows the metric to account for word boundaries and restrict inter-word jumps, making it more linguistically grounded.
- Coverage Penalty: EED introduces a penalty for characters that are aligned more than once or not at all in the candidate translation. This is quantified by
v, the sum of visits for characters visited multiple times, scaled by a factorρ. This penalty helps to penalize word repetitions in the reference more effectively than just jump costs.
The metric is defined as a combination of edit operations (e), jumps (j), and the coverage penalty (v), normalized by the reference length and a coverage penalty term. The authors also detail how they optimized the parameters (α for jumps, ρ for coverage, and deletion cost) by searching for values that maximize human correlation on WMT datasets.
Datasets
The paper primarily uses data from the WMT (Workshop on Machine Translation) shared tasks, specifically the newstest2018 test set. This dataset consists of approximately 3000 sentences per language pair and provides one reference translation per source sentence. The evaluation is performed at both the segment level (using relative ranking, DARR) and the system level (using direct assessment, DA).
Key Insights and Ideas
- Jumps are powerful but need constraints: The paper highlights that while jumps offer flexibility in aligning translations, restricting them to word boundaries (blanks in the reference) is crucial for maintaining linguistic relevance and preventing overly permissive alignments.
- Coverage penalty addresses repetition: The introduction of a coverage penalty is a clever way to penalize redundant word usage in translations, a common issue that simpler edit distance metrics might overlook.
- Parameter tuning is critical for correlation: The authors demonstrate that careful tuning of parameters like deletion cost, jump cost (
α), and coverage cost (ρ) significantly impacts the metric's correlation with human judgments. - Character-level operations can be effective: EED operates at the character level, which, as noted, can sometimes lead to better human correlation than word-level metrics, especially when combined with intelligent extensions like jumps.
Key Results
- Competitive Human Correlation: EED achieves competitive human correlation scores, ranking among the top metrics on segment-level evaluations (DARR) and performing particularly well in the "out of English" direction on system-level evaluations (DA). For instance, on system-level DA, EED is the second best after RUSE for the "to English" direction.
- Speed and Efficiency: EED is noted as being fast to compute, second only to BLEU among the tested metrics, making it suitable for rapid development cycles. Its memory usage is also comparable to other metrics, with shift-based metrics requiring more.
- Parameter Optimization: The paper shows that optimal parameter values for deletion cost (0.2), jump cost (
α= 2.0), and coverage cost (ρ= 0.3) were found to maximize human correlation.
Limitations and Future Work
- Normalization Strategy: While EED uses a simple and efficient method of normalizing by the reference length and capping the score at 1.0, the authors acknowledge that other normalization strategies were explored but led to a decline in correlation or were more complex.
- Future Work: The authors suggest that a more in-depth analysis of the field is required, and specifically mention that the relationship between shifts and jumps will be investigated further in the future.
Summary updated