TL;DR
This paper explores how to improve machine translation performance prediction (MTPP) by stacking multiple "expert" prediction models. They found that combining word-level features with sentence/document-level features significantly boosts performance on training data but surprisingly leads to a decrease in test set performance, suggesting a potential overfitting issue.
What Problem Does This Paper Address?
Predicting the quality of machine translation (MT) without human references is a crucial task for various applications, such as filtering low-quality translations or providing real-time feedback to users. This is known as Machine Translation Performance Prediction (MTPP). The challenge lies in developing models that can accurately estimate translation quality using only the source text and the machine-translated output. Existing methods often rely on a single model or a simple ensemble. This paper investigates a more sophisticated approach: "stacking" multiple prediction models, or "experts," to create a more robust and potentially higher-performing predictor. They aim to see if a richer set of features, combining different levels of granularity (word, sentence, document), can lead to better MTPP.
Methods
The core of this work revolves around Referential Translation Machines (RTMs) and a stacking approach. RTMs are used to generate features that capture the relationship between training and test data, aiding in prediction. The authors experiment with various machine learning models as "experts," including ridge regression, kernel ridge regression, k-nearest neighbors, support vector regression, AdaBoost, gradient tree boosting, Gaussian process regressor, extremely randomized trees, and multi-layer perceptrons.
The key novelty lies in their stacking strategy. They combine features extracted from word-level predictions (like identifying specific translation errors) with sentence- and document-level features. This is done by training a "meta-model" on the outputs of these individual expert models. They also explore two main combination strategies:
- Mixture of Experts (MIX): This involves a weighted average of predictions, where weights are determined by the performance of individual experts on the training set.
- Stacking: This is a hierarchical approach where the predictions of base models are used as features for a second-level model.
Datasets
The experiments were conducted on datasets from the WMT19 Quality Estimation Task (QET19). This task includes subtasks for English-German, English-Russian, and English-French machine translation. The paper specifies the number of sentences in the training and test sets for each task, as well as the number of "interpretants" (data points used by RTMs) for each.
Key Insights and Ideas
- The Double-Edged Sword of Granular Features: Combining word-level error prediction features with sentence/document-level features significantly improves performance on the training set. This suggests that these detailed features are highly informative for the model when it has seen the data before.
- The Test Set Drop-off: Despite the training set gains, this combined feature set leads to a decrease in performance on the test set. This is a critical finding, strongly indicating overfitting. The model is learning patterns specific to the training data that don't generalize well to unseen translations.
- Stacking vs. MIX: The paper suggests that stacking generally achieves better results than the MIX approach, implying that a more complex meta-model can better leverage the diverse predictions of the base experts.
- RTMs as Feature Generators: RTMs are presented as a robust way to generate context-aware features for MTPP, adaptable to different language pairs and translation systems (SMT vs. NMT).
Key Results
The paper presents results in Tables 2, 3, and 5. While specific percentage improvements are not highlighted as headline figures, the tables show various metrics like Pearson's correlation (rp), Mean Absolute Error (MAE), and Root Mean Squared Error (RMSE).
- For instance, in Table 3, for the
en-de SMTtask, adding word tags to sentence-level predictions improves therpfrom 0.4386 to 0.8675 and reduces MAE from 0.1368 to 0.1368 (though this specific MAE value seems to be a typo in the OCR, likely intended to be a reduction). The paper states that the addition of tagging features improves training results significantly. - However, the test set results (Table 5) show a more nuanced picture. For
en-de, adding word tags to sentence-level predictions results in a lowerrp(0.4617 vs. 0.5718 for the top result) and higher MAE (0.1176 vs. 0.187). This confirms the overfitting observation.
Limitations and Future Work
- Overfitting: The most significant limitation identified is the overfitting observed when combining word-level features with sentence/document-level features. The high correlation of word-level features with the target metric on the training set is cited as a reason.
- Feature Selection: The paper mentions filtering results based on certain criteria and suggests that further feature selection might be beneficial.
- Future Directions: The authors implicitly suggest exploring methods to mitigate overfitting, such as more robust regularization techniques or different feature engineering strategies that generalize better. They also highlight the potential of RTMs for language-independent and high-performance MTPP.
Summary updated