Abstract
Current re-ranking algorithms for machine translation rely on log-linear models, which have the potential problem of underfitting the training data. We present BoostedMERT, a novel boosting algorithm that uses Minimum Error Rate Training (MERT) as a weak learner and builds a re-ranker far more expressive than log-linear models. BoostedMERT is easy to implement, inherits the efficient optimization properties of MERT, and can quickly boost the BLEU score on N-best re-ranking tasks. In this paper, we describe the general algorithm and present preliminary results on the IWSLT 2007 Arabic-English task.
The Modeling Gap in Machine Translation
In Machine Translation (MT), a decoder typically generates an N-best list of candidate translations. A second-pass re-ranker then selects the final output from this list. Despite the presence of high-quality "oracle" hypotheses in these lists, standard re-rankers often fail to select them.
The authors argue that this failure is primarily a modeling problem rather than an optimization one. Most systems utilize log-linear models with a small number of features (typically around 10). These models lack the expressive power to distinguish the best translations from mediocre ones, creating a performance ceiling regardless of how well the weights are optimized.
BoostedMERT: Ensembling for Expressivity
To overcome the limitations of linear models, the authors introduce BoostedMERT. This approach adapts the concept of boosting—combining multiple "weak" learners into a single "strong" ensemble—to the MT re-ranking task.
- MERT as a Weak Learner: Minimum Error Rate Training (MERT), the standard optimization tool for MT, serves as the base learner.
- Weighted Distribution: The algorithm maintains weights for each sentence in the training set. Sentences where the current model performs poorly (i.e., picks a hypothesis far from the oracle) receive higher weights in subsequent iterations to force the model to focus on difficult cases.
- Non-Linear Voting: Instead of a simple linear combination of features, the final ranker uses a voting scheme based on reciprocal ranks. This introduces the non-linearity necessary to capture complex linguistic nuances that a simple dot product would miss.
Experimental Setup
The method was evaluated on the IWSLT 2007 Arabic-to-English task. The system re-ranked N-best lists of size $N=2000$ using 9 standard MT features. Training and development were conducted on the IWSLT Dev4 and Dev5 sets, respectively, with results reported on the official 2007 Evaluation set.
Results and Discussion
Experimental results demonstrate that BoostedMERT consistently outperforms the standard MERT baseline by leveraging the expressive power of the ensemble.
As shown in the results table, standard MERT achieved a BLEU score of 41.2 on the evaluation set. BoostedMERT improved this to 42.0 (Selected BLEU), representing a +0.8 gain. Notably, the "Best BLEU" observed during boosting iterations reached 43.7, indicating a potential +2.5 point improvement if iteration selection is further refined. This suggests that the boosting framework successfully moves the performance closer to the oracle ceiling than linear methods ever could with the same feature set.
Conclusion and Future Work
BoostedMERT provides a practical path toward more expressive MT models without requiring entirely new feature sets. However, several challenges remain:
- Iteration Selection: There is a significant gap between the best possible iteration and the one selected by the development set, necessitating more robust stopping criteria.
- Data Sensitivity: Improvements tend to diminish as training set size increases, suggesting the need for more diverse weak learners.
- Weak Learner Strength: Because MERT is already a strong optimizer, the traditional theoretical benefits of boosting (which assume weak learners) may not fully materialize, leading to diminishing returns over time.
Summary updated