Abstract
Recent advancements in Statistical Machine Translation (SMT) have explored tuning systems with large numbers of sparse features. However, practical examples of beneficial sparse features for phrase-based systems remain limited. This paper addresses reordering—a traditional weakness of phrase-based translation—by introducing sparse features into a hierarchical reordering model (HRM). By coupling phrase orientation with frequent words or word clusters, we demonstrate significant improvements in translation quality, achieving gains of up to 1.2 BLEU points in Chinese-English and 1.8 BLEU points in Arabic-English tasks. Our results indicate that sparse decoder features, optimized directly for BLEU, significantly outperform traditional maximum entropy models trained on word-aligned bitext.
Introduction: The Reordering Challenge
In Phrase-Based Statistical Machine Translation (SMT), "reordering"—the process of determining the correct word order in the target language—is a persistent challenge. Standard models typically rely on "lexicalized reordering," which calculates the probability of orientations (e.g., Monotone, Swap, or Discontinuous) based on their relative frequencies in a training corpus.
A major hurdle is data sparsity. Most phrase pairs appear infrequently, leaving the model with insufficient statistics to handle new or rare phrases. While researchers have attempted to mitigate this using Maximum Entropy (MaxEnt) models to "smooth" these statistics across millions of sentences, these models often fail to deliver proportional gains in actual translation quality. This is primarily due to an "objective mismatch": MaxEnt models optimize for predicting training data (likelihood), which does not always correlate with producing better translations (BLEU).
Methodology
This research compares three distinct approaches to the Hierarchical Reordering Model (HRM):
- Standard HRM (Baseline): Employs relative frequencies from word-aligned bitext, smoothed using MAP (Maximum A Posteriori) techniques.
- Maximum Entropy HRM: A complex model trained on millions of bitext examples. It uses features like the first and last words of a phrase and word clusters to predict reordering orientations.
- Sparse Feature HRM: The proposed approach integrates thousands of simple indicator features directly into the translation decoder. These features (e.g., "Does this phrase start with word X and result in a Swap?") are tuned on a small development set (a few thousand sentences) using an optimizer like MIRA to directly maximize the BLEU score.
The sparse features utilize three levels of representation: lexical (actual words), 50-word clusters, and 20-word clusters. This hierarchy allows the model to generalize from specific lexical items to broader grammatical categories.
Experimental Setup
Experiments were conducted on two large-scale translation tasks:
- Arabic-to-English: Trained on 1.49 million sentence pairs.
- Chinese-to-English: Trained on 3.5 million sentence pairs. Evaluation utilized NIST MT evaluation data, with benchmarks including mt06, mt08, and mt09.
Key Insights
The "Decoder in the Loop" Advantage
The primary finding is that optimizing directly for translation quality (BLEU) on a small, representative dataset is more effective than optimizing for likelihood on a massive one. Sparse features allow the optimizer to observe the decoder's actual behavior and correct specific errors in situ.
Objective Mismatch
The study highlights a disconnect between model perplexity and translation performance. As shown in the comparison between MaxEnt and relative frequency models, increasing training data significantly improves a model's ability to predict alignments (lower perplexity), but these improvements do not consistently translate into higher BLEU scores.
Simplicity and Context
Effective reordering does not require deep linguistic or syntactic parsing. Simply tracking the "corners" of a phrase—the first and last words or clusters—provides a sufficient signal for the optimizer. Furthermore, these features are most effective when they have access to the decoder's "stack" context, allowing them to account for the preceding translated content.
Results and Performance
- Significant BLEU Gains: The Sparse HRM outperformed the baseline by 1.2 BLEU points in Chinese-English and 1.8 BLEU points in Arabic-English.
- Superiority over MaxEnt: In direct comparisons, sparse features tuned for BLEU consistently outperformed MaxEnt models, even those trained on significantly more data.
- Robustness: The approach provided a consistent boost (0.4 to 1.1 BLEU) even when integrated into high-performance SMT systems already utilizing thousands of other features.
Conclusion and Future Directions
While the Sparse HRM offers substantial quality improvements, it introduces a computational overhead, running approximately four times slower than the baseline decoder in experimental settings. Future work may focus on pre-calculating weights to mitigate this latency. Additionally, exploring "transition-based" features that look deeper into the reordering stack could further enhance long-distance reordering capabilities.
Summary updated