TL;DR
This paper introduces a novel machine learning approach for automatically ranking multiple machine translation (MT) outputs for a given source sentence, without relying on reference translations. By decomposing the ranking problem into pairwise comparisons and using "soft recomposition" that weighs decisions by their confidence, the system achieves significantly better correlation with human judgments than previous methods and even outperforms some state-of-the-art reference-based metrics.
What Problem Does This Paper Address?
As machine translation (MT) systems become more prevalent, there's a growing need to quickly and accurately assess the quality of their output in real-time, without the luxury of having human-created reference translations. Traditional MT evaluation metrics often require these references, making them unsuitable for many practical applications. Furthermore, while some "quality estimation" methods aim to predict absolute quality scores, this paper focuses on a more nuanced task: ranking multiple translation candidates for the same source sentence. This is crucial for applications like system selection or choosing the best translation from a set of options. The challenge lies in building a system that can reliably mimic human judgment in this comparative ranking task, especially when dealing with subtle differences in translation quality and the potential for ties.
Methods
The core of the proposed method is a machine learning approach that treats the ranking problem as a series of pairwise comparisons. For each source sentence, all possible pairs of its translated outputs are generated. A binary classifier is then trained on these pairs, learning to predict whether one translation is better than another.
The key innovation lies in how the pairwise decisions are recomposed into a final ranking. Instead of simply counting "wins" (a "hard recomposition"), the paper introduces "soft recomposition." This method weighs each pairwise decision by its confidence probability. This means that a win from a highly confident prediction contributes more to the final rank than a win from a less confident prediction. This probabilistic weighting helps to eliminate ranking ties and provides a more nuanced ranking.
The system utilizes a variety of linguistic features extracted from the source and translated sentences, including:
- Count-based features: token counts, character counts per token, unknown words.
- Parsing statistics: features derived from Probabilistic Context Free Grammars (PCFG) parsing, such as parse log-likelihood and confidence scores.
- Tree label counts: ratios of specific grammatical structures (e.g., NPs, VPs) between source and target.
- Language checking: scores from automatic grammar and style checkers.
- Language-model probabilities: n-gram probabilities indicating fluency.
- Contrastive evaluation scores: scores from existing automatic metrics like METEOR, using other translations as references.
Several machine learning algorithms were explored for the pairwise classification, including Naïve Bayes, k-Nearest Neighbors (k-NN), and Logistic Regression, with Logistic Regression ultimately showing the best performance.
Datasets
The study utilizes data from the Workshops on Machine Translation (WMT) evaluation tasks. Specifically, human-annotated rankings from the German-to-English translation tasks of years 2008, 2009, 2010, and 2011 were used. The data consists of human judgments organized in rankings of up to five sentences. For the test set, multiple rankings of the same source sentence were aggregated, and ties were removed.
Key Insights and Ideas
- Pairwise comparison is a powerful simplification: Decomposing the complex ranking task into simpler binary decisions makes it amenable to standard machine learning classifiers.
- Confidence matters: Soft recomposition is key: Simply counting pairwise wins (hard recomposition) leads to too many ties. Weighing these wins by the classifier's confidence probability (soft recomposition) significantly improves ranking accuracy and reduces ties. This suggests that the certainty of a prediction is as important as the prediction itself.
- Reference-free ranking can compete: The proposed system, which does not require reference translations, achieves performance comparable to or better than established reference-based metrics like METEOR and Levenshtein distance. This has significant implications for real-time MT evaluation.
- Linguistic features are crucial: A rich set of features, particularly those derived from parsing and contrastive evaluation, are essential for capturing subtle aspects of translation quality.
Key Results
The most successful system, using Logistic Regression with a carefully selected feature set (including unknown words, sentence length, contrastive METEOR, and parse statistics), achieved a correlation with human judgments (Kendall's tau) of up to 0.30. This is a significant improvement over previous reference-free methods and, importantly, is at least 35% higher than that of standard reference-based metrics like smoothed-BLEU, METEOR, and Levenshtein distance on a separate test set. The soft recomposition method improved correlation numbers by 40-80% in the best cases, while also drastically reducing the number of ties.
Limitations and Future Work
The authors acknowledge that repetitive human rankings of the same systems sometimes disagree, indicating noise in the data. While they explored various feature sets and classifiers, the paper mentions that the approach might have stalled at local maxima due to the incremental experimental strategy. Future work could involve exploring more sophisticated tie-handling mechanisms beyond soft recomposition, such as introducing a third class or using cascaded classifiers. Further investigation into the interpretability of the learned features and their correlation with specific types of translation errors could also be beneficial.
Summary updated