TL;DR
This paper introduces two novel methods for creating smaller, more efficient grammars for hierarchical phrase-based machine translation (Hiero). By using combinatorial optimization or a Bayesian model with Variational Bayes, they achieve significant grammar size reductions without sacrificing translation quality, and even improve performance on low-resource languages.
What Problem Does This Paper Address?
Hierarchical phrase-based statistical machine translation (Hiero) has shown strong performance, but a major drawback is the enormous size of the trained translation models. These large models lead to slower decoding and more search errors. The authors aim to address this by developing methods to extract compact Hiero grammars, reducing model size while maintaining or improving translation quality.
Methods
The paper proposes two main approaches:
- Combinatorial Optimization (Greedy Approach): This method frames the problem as finding a minimal set of translation rules that can cover all possible derivations for initial phrase pairs. It's formulated as a minimum set cover problem on a tripartite graph representing phrase pairs, derivations, and rules. A greedy algorithm is used to select rules that cover the most derivations.
- Bayesian Model with Variational Bayes (VB): This approach treats Hiero grammar extraction as a Bayesian inference problem. They develop a generative model for phrase pairs and use Variational Bayes (VB) for inference to learn a posterior distribution over grammar parameters and derivations. This method learns a compact grammar that is not necessarily minimal but aims for a good trade-off between size and performance.
Datasets
The experiments were conducted on three language pairs:
- Arabic-English: Using the ISI Ar-En corpus.
- English-Spanish: Using WMT10 data combined with UN data.
- Korean-English: Using the University of Rochester corpus.
The paper does not introduce new benchmarks but uses standard datasets for evaluating machine translation systems.
Key Insights and Ideas
- 1NT vs. 2NT Grammars: The research shows that grammars with only one non-terminal (1NT) can achieve comparable translation quality to those with two non-terminals (2NT), suggesting that the expressive power isn't significantly lost. This is crucial for the Bayesian model, which naturally produces 1NT rules.
- VB for Better Parameter Estimation: The Bayesian model with Variational Bayes leads to a "sharper distribution" of probabilities, effectively learning better parameters. This means it can better distinguish between high-quality and low-quality rules, especially in low-resource scenarios.
- Compactness without Quality Loss: Both proposed methods demonstrate that significant grammar size reduction (up to 57.6%) is achievable without negatively impacting translation quality, and in some cases, even improving it.
- Greedy Approach's Weakness: The greedy combinatorial approach, while aiming for minimality, often misses crucial hierarchical rules, leading to poorer translation performance compared to the VB approach. This suggests that simply covering derivations isn't enough; the quality of the rules matters.
- Unaligned Words: The VB model's ability to handle unaligned source words by attaching them at multiple positions in derivations, unlike the original Hiero's more restrictive approach, contributes to its robustness.
Key Results
- Model Size Reduction: The VB approach achieved model size reductions ranging from 17.8% to 57.6% across different language pairs and pruning strategies.
- Translation Quality: The VB model achieved comparable or better BLEU scores than the original Hiero models, particularly showing significant improvements for the low-resource Korean-English pair.
- Speed Improvement: Compact grammars, especially the pruned VB grammars, resulted in faster decoding speeds, ranging from 10-35% faster compared to original Hiero models.
- Greedy Approach Performance: The greedy approach generally resulted in lower BLEU scores compared to the VB and original Hiero models, despite achieving smaller model sizes.
Limitations and Future Work
- VB Inference Complexity: The VB inference is computationally intensive, especially for large datasets. The authors filtered phrase pairs for Arabic-English and English-Spanish to make it tractable.
- 2 Non-terminals: The current VB model produces 1NT rules, while Hiero traditionally uses 2NT rules. Future work aims to extend the model to allow 2 non-terminals.
- Large-Scale Datasets: The authors plan to apply their Variational Bayes approach to even larger-scale SMT datasets.
- Model Selection: The paper mentions using an informative prior for q(θ) to push towards sparsity, but doesn't explicitly detail a free energy criterion for model selection as seen in some other VB applications.
Summary updated