TL;DR
This paper proposes a novel method to improve automatic machine translation (MT) evaluation by filtering automatically generated "pseudo-references." By using BERT to check if these pseudo-references are paraphrases of the human-written "gold reference," the authors show that this filtering significantly improves the correlation between automatic evaluation scores and human judgment, outperforming previous methods.
What Problem Does This Paper Address?
Evaluating the quality of machine translation (MT) automatically is a persistent challenge. Traditional metrics like BLEU rely on n-gram overlap with human-written reference translations (gold references). However, this approach struggles when a good translation has a different surface form than the reference. To address this, researchers have explored using multiple references, including "pseudo-references" generated by other MT systems. While this can improve evaluation, the quality of these pseudo-references is often poor, leading to worse correlation with human judgment than using a single gold reference. This paper tackles the problem of leveraging pseudo-references effectively by ensuring they are semantically equivalent to the gold reference, even if their wording differs.
Methods
The core of the proposed method is a filtering step applied to pseudo-references generated by off-the-shelf MT systems.
- Pseudo-Reference Generation: Outputs from readily available MT systems (Google Translate and Bing Translator in this case) are used to generate pseudo-references for source sentences.
- Paraphrase Filtering with BERT: The key innovation is using a pre-trained BERT model, fine-tuned on the Microsoft Research Paraphrase Corpus (MRPC), to assess the paraphrasability of each pseudo-reference against the gold reference. BERT, a powerful transformer-based language model, is well-suited for understanding sentence-level semantic similarity. The model outputs a probability score indicating how likely a pseudo-reference is a paraphrase of the gold reference.
- Filtering Threshold: Pseudo-references with a BERT paraphrase probability score above a certain threshold (0.5 in this study) are retained.
- Evaluation Metric: The filtered pseudo-references are then combined with the single gold reference, and the Sentence-BLEU (SentBLEU) score is calculated using this expanded set of references. SentBLEU is a smoothed version of BLEU designed for sentence-level evaluation.
Datasets
- MRPC (Microsoft Research Paraphrase Corpus): Used for fine-tuning the BERT model to classify sentence pairs as paraphrases or not.
- WMT 2016 and WMT 2017 Segment-Level Evaluation Datasets: These datasets, covering Czech-English (cs-en), German-English (de-en), Finnish-English (fi-en), and Russian-English (ru-en) language pairs, were used for evaluating the proposed method. They include source sentences, system outputs, gold references, and human evaluation scores.
Key Insights and Ideas
- Semantic Equivalence is Key: The paper strongly suggests that simply having more references isn't enough; the references must be semantically equivalent to the gold standard. Poor quality pseudo-references can actively harm evaluation.
- BERT as a Paraphrase Detector: Leveraging BERT's ability to understand nuanced semantic relationships between sentences proves highly effective for filtering out irrelevant or misleading pseudo-references.
- Balancing Surface Form and Meaning: The method successfully identifies references that have different surface forms but retain the original meaning, which is crucial for robust MT evaluation.
- Improved Correlation: The filtered pseudo-references lead to a demonstrably higher correlation with human evaluation scores compared to using only a single gold reference or unfiltered pseudo-references.
Key Results
The paper reports significant improvements in Pearson correlation coefficients between automatic and human evaluation scores across multiple language pairs and years.
- WMT 2016: The proposed method (single reference + filtered references (BERT)) achieved the highest correlation for cs-en (0.589), de-en (0.519), and fi-en (0.572), outperforming baselines.
- WMT 2017: Similarly, the method showed the best performance for cs-en (0.555), de-en (0.580), and fi-en (0.671).
- While the ru-en language pair in WMT 2016 was an exception where the proposed method did not outperform all baselines, the overall trend is positive.
- The paper also notes that filtering by Maximum Alignment Similarity (MAS) did not yield consistent results.
Limitations and Future Work
- Language Pair Specificity: While the method shows promise across several language pairs, its performance might vary for languages with significantly different linguistic structures or for MT systems with very distinct error patterns.
- Threshold Sensitivity: The choice of the paraphrase probability threshold (0.5) could impact the results. Further investigation into optimal threshold selection for different scenarios might be beneficial.
- Computational Cost: Using BERT for filtering adds a computational overhead compared to simpler methods.
- Future Work: The authors suggest that this approach can be applied to various MT evaluation metrics that can benefit from multiple references. Further exploration into different filtering strategies and their impact on evaluation metrics is also a potential avenue.
Summary updated