Abstract: There has been a proliferation of recent work on SMT tuning algorithms capable of handling larger feature sets than the traditional MERT approach. We analyze a number of these algorithms in terms of their sentence-level loss functions, which motivates several new approaches, including a Structured SVM. We perform empirical comparisons of eight different tuning strategies, including MERT, in a variety of settings. Among other results, we find that a simple and efficient batch version of MIRA performs at least as well as training online, and consistently outperforms other options.
Introduction
In Statistical Machine Translation (SMT), "tuning" is the critical process of optimizing feature weights—such as word frequency or translation probability—to maximize a translation quality metric, typically BLEU. For over a decade, Minimum Error Rate Training (MERT) served as the industry standard. However, MERT faces a significant "scaling wall," becoming mathematically unstable when handling more than approximately 30 features.
As research has shifted toward "sparse features" involving thousands of fine-grained indicators, MERT has become a bottleneck. While online methods like MIRA were developed to address this, they are notoriously difficult to parallelize across high-performance computing clusters. This paper explores a middle ground: batch-based optimization strategies that can handle thousands of features while maintaining the easy-to-parallelize infrastructure researchers already use for MERT.
Tuning Methodologies
We evaluate several existing strategies and introduce three new approaches, all unified by their use of sentence-level loss functions.
Existing Methods
- MERT: The traditional baseline, limited by its inability to scale to large feature sets.
- Online MIRA: Updates weights sentence-by-sentence using "hope-fear" logic, moving weights toward high-BLEU/high-model-score translations and away from low-BLEU/high-model-score ones.
- Pairwise Ranking Optimization (PRO): Recasts tuning as a classification problem by sampling pairs of translations and learning to rank the superior candidate higher.
- Minimum Risk (MR): A probabilistic approach that minimizes the expected error across the entire distribution of possible translations.
Proposed Approaches
- Batch MIRA (k-best and Lattice): Our primary innovation. It adapts the "hope-fear" update logic of MIRA to a fixed set of translations (k-best lists or lattices) in a batch. This allows the decoder to run in parallel while the optimizer runs efficiently on a single machine.
- Structured SVM: A direct optimizer that minimizes a "hinge loss" specifically designed for the structured output of translation tasks.
Experimental Setup
The methods were tested across three language pairs: French-English (Hansard Corpus), English-French (Hansard), and Chinese-English (NIST). We evaluated three distinct feature-set sizes: Small (~7 features), Medium (~18 features), and Big (up to 6,848 sparse features).
Key Results and Insights
The Efficiency of Batch MIRA
Perhaps the most significant finding is that Batch Lattice MIRA (labeled as lb-mira in figures) performed as well as, or better than, its online counterpart. It emerged as the most consistent winner, securing the top score in all "Medium" feature settings and the majority of "Big" settings. Furthermore, Batch MIRA was an order of magnitude faster to train than the online version because it avoided the heavy inter-process communication required to keep online weights synchronized.
Lattices and Regularization
Using lattices—which represent a massive number of possible translations—instead of simple k-best lists significantly improved the stability and performance of MIRA-based tuners. Additionally, direct optimizers like the Structured SVM allowed for explicit regularization. While MIRA has a "natural" regularization through small step sizes, the explicit $\lambda$ parameter in SVMs provided finer control over overfitting, particularly in the complex Chinese-English "Big" feature setting, where the SVM actually outperformed Batch MIRA.
Conclusion
Our results demonstrate that the complexity of true online learning is not necessary for state-of-the-art SMT tuning. Batch Lattice MIRA provides a simple, fast, and highly effective alternative that scales gracefully to thousands of features. While sparse features currently offer modest BLEU gains, the ability to optimize them efficiently opens the door for more complex feature engineering in future SMT research.
Summary updated