Online Large-Margin Training of Syntactic and Structural Translation Features

David Chiang, Yuval Marton, Philip Resnik

Abstract

Statistical Machine Translation (SMT) systems have traditionally been constrained by the "MERT bottleneck," where the standard optimization method, Minimum Error Rate Training, fails when scaled beyond a few dozen features. This paper presents a transition to the Margin Infused Relaxed Algorithm (MIRA), an online large-margin learning framework capable of optimizing high-dimensional feature spaces. By introducing parallelized MIRA and novel features for soft syntactic constraints and structural distortion, the authors demonstrate a significant 2.6 BLEU point gain in Arabic-to-English translation tasks, proving that complex linguistic signals can be effectively integrated into hierarchical translation models.

Introduction

In the mid-2000s, the progress of Statistical Machine Translation (SMT) was hindered by the limitations of Minimum Error Rate Training (MERT). While reliable for small feature sets, MERT becomes unstable when handling more than 20 to 30 weights. This forced researchers to manually prune potentially valuable linguistic signals, leaving powerful features on the cutting room floor. This paper addresses this bottleneck by adapting MIRA for translation, allowing for the simultaneous training of dozens of features and enabling the use of more sophisticated syntactic and structural models.

The MIRA Framework and Oracle Selection

Unlike MERT, which performs batch optimization on the entire tuning set, MIRA is an online algorithm that updates model weights on a sentence-by-sentence basis. To make this viable for translation, the authors introduced several key innovations:

  1. Parallelized Training: Multiple processors decode sentences independently and share hypothesis translations, ensuring training speeds comparable to or faster than MERT.
  2. Balanced Oracle Selection: A critical challenge in online learning for SMT is selecting the "oracle" (the target translation for weight updates). Simply choosing the candidate with the highest BLEU score can lead to unstable updates if that candidate has a very low model score. The authors proposed a balanced approach using a parameter $\mu$ to find an oracle that maximizes a combination of BLEU and model scores, stabilizing the learning process.
Figure 1: Distribution of translation candidates showing the trade-off between Model Score and BLEU score for different values of the balance parameter $\mu$.

Structural and Syntactic Features

The increased capacity of MIRA allowed the authors to move beyond the "syntax-light" nature of the standard Hiero model. They introduced two major sets of features to capture linguistic nuances:

  • Soft Syntactic Constraints (34 features): These features utilize a source-side parser to reward or penalize translations based on whether they respect syntactic boundaries (e.g., Noun Phrases or Verb Phrases) from the source text.
  • Structural Distortion (34 features): These features track the reordering of non-terminals based on their span length, addressing the lack of a formal distortion model in hierarchical systems and penalizing or rewarding long-distance movements based on phrase size.
Figure 6: Examples of (a) reordered and (b) non-reordered rules within the hierarchical phrase-based translation framework.

Experimental Results

The proposed system was evaluated on Arabic-to-English translation using NIST evaluation data. The results confirmed that MIRA is not just a replacement for MERT but an enabler for more complex modeling:

  • Performance Gain: With the addition of fine-grained syntactic and distortion features (totaling 56 weights), the MIRA-tuned system achieved a 2.6 BLEU point improvement over the MERT baseline on the NIST 2006 test set.
  • MERT Failure: Attempts to train the same 34 syntactic features using MERT resulted in no improvement, highlighting the reality of the MERT bottleneck.
  • Efficiency: MIRA converged in approximately 8 passes through the tuning data, proving to be as efficient as MERT while delivering significantly higher final quality.

Conclusion

The shift from batch tuning to online large-margin training represents a significant step forward for SMT. By overcoming the MERT bottleneck, researchers can now incorporate fine-grained linguistic and structural features that were previously computationally inaccessible. This work demonstrates that syntax and structural reordering are vital components of translation quality when optimized within a framework capable of handling high-dimensional feature spaces.

Summary updated

Abstract

Minimum-error-rate training (MERT) is a bottleneck for current development in statistical machine translation because it is limited in the number of weights it can reliably optimize. Building on the work of Watanabe et al., we explore the use of the MIRA algorithm of Crammer et al. as an alternative to MERT. We first show that by parallel processing and exploiting more of the parse forest, we can obtain results using MIRA that match or surpass MERT in terms of both translation quality and computational cost. We then test the method on two classes of features that address deficiencies in the Hiero hierarchical phrase-based model: first, we simultaneously train a large number of Marton and Resnik's soft syntactic constraints, and, second, we introduce a novel structural distortion model. In both cases we obtain significant improvements in translation performance. Optimizing them in combination, for a total of 56 feature weights, we improve performance by 2.6 Bleu on a subset of the NIST 2006 Arabic-English evaluation data.
BibTeX
@inproceedings{chiang2008online,
  author = {David Chiang and Yuval Marton and Philip Resnik},
  title = {{Online Large-Margin Training of Syntactic and Structural Translation Features}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2008},
}
Figure