Abstract: Sweta Agrawal, George Foster, Markus Freitag, Colin Cherry. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
2021.
Introduction
Traditional machine translation (MT) evaluation relies heavily on reference-based metrics like BLEU, which compare a system's output to a human-written translation. While standard, this approach suffers from high costs, narrowness (penalizing valid but non-matching translations), and poor scalability to new domains or low-resource languages.
This paper explores reference-free evaluation, where a model assesses quality using only the source text and the system's output. The authors demonstrate that by scaling a multilingual MT model to a "Massive" size and refining scoring techniques, reference-free evaluation can achieve performance on par with industry-standard metrics, effectively turning a translation model into a reliable judge.
Methods and Model Scaling
The authors build upon Prism, a method that uses the internal probabilities of a multilingual MT model to score translations. A key contribution of this work is investigating the impact of model capacity on evaluation accuracy. They tested three Transformer architectures of increasing size, culminating in a "Massive" 1.8-billion parameter model.
Beyond scaling, the researchers introduced several scoring refinements:
- Scoring Regularization: Utilizing Monte-Carlo dropout and subword regularization to produce smoother, more robust quality scores.
- Aggregation Strategies: Experimenting with median and minimum token scores rather than simple averages.
- Confidence Thresholds: A novel mapping technique that prioritizes the model's most certain predictions (converting high/low probabilities to +1/-1) while ignoring uncertain "middle" probabilities.
Key Insights: The Evaluator Paradox
One of the paper's most significant findings is the "Good Judge vs. Good Translator" Paradox. A model does not need to be the state-of-the-art translator to be an excellent evaluator. The authors found that their evaluating model was often "middle-of-the-road" in terms of its own translation quality but could still accurately rank systems that outperformed it.
Furthermore, the study addresses the "proximity bias" concern—the fear that a model might unfairly favor systems that produce output similar to its own. The results showed no evidence of such bias; the metric remained objective even when judging systems with vastly different architectures or training data than the evaluator.
Results and Performance
The "Massive" Prism model achieved an average system-level correlation of 0.883 with human judgments. While this raw correlation is slightly lower than BLEU's 0.911, the reference-free approach reached "statistical parity" in pairwise comparisons, meaning it is just as effective as BLEU at deciding which of two systems is superior.
In head-to-head system rankings, the Massive Prism model correctly ranked systems 88% of the time, marginally outperforming BLEU's 87%. Notably, the performance of this reference-free approach improves steadily as the test set size increases, making it particularly well-suited for large-scale evaluations of web-crawled data where human references are unavailable.
Limitations and Future Work
Despite its successes, the reference-free approach faces challenges in languages with low representation in training data (e.g., English-Gujarati) and remains sensitive to domain shifts. Additionally, the computational cost of running a 1.8B parameter model for evaluation is significantly higher than calculating simple metrics like BLEU. Future research may focus on optimizing model architectures specifically for the evaluation task rather than repurposing standard translation models.
Summary updated