TL;DR
This paper introduces a novel method for "pruning" statistical machine translation (SMT) models, specifically by removing less useful phrase pairs. By analyzing how often phrase pairs are actually used during translation, they achieve significant reductions in model size with minimal impact on translation quality, making SMT more feasible for memory-constrained devices.
What Problem Does This Paper Address?
Modern statistical machine translation (SMT) systems rely heavily on large "translation models" that store probabilities for millions of phrase pairs (e.g., "how to say X in English"). While effective, these models can be enormous, posing a significant challenge for deployment on devices with limited memory, such as smartphones or PDAs. Existing pruning methods, like simple probability thresholds or limiting the number of translations per source phrase, are often too blunt and can degrade translation quality significantly. The authors aim to find a more nuanced way to prune these models, reducing their size while preserving translation performance.
Methods
The core of the paper's contribution is a new pruning approach inspired by "Optimal Brain Damage" from neural networks. Instead of relying on static probabilities or counts, they propose to measure the actual usage of phrase pairs during the translation process.
Here's how it works:
- Translate a Large Corpus: The existing SMT system is used to translate a substantial amount of in-domain text.
- Collect Usage Statistics: For each phrase pair in the translation model, two counts are collected:
c(phrase pair): How often the phrase pair was considered during the decoding process (i.e., appeared in the translation lattice).u(phrase pair): How often the phrase pair was actually used in the final, chosen translation hypothesis.- Calculate Relevance Score: A scoring function is used to combine these counts, giving more weight to actual usage (
u) than mere consideration (c). The proposed score islog(c(phrase pair) + 1) * (u(phrase pair) + 1). The logarithm oncdampens its influence, whileudirectly reflects how often a phrase pair contributes to a final translation. - Prune: Phrase pairs are sorted by this relevance score, and the lowest-scoring ones are removed.
This method is "usage-based" and doesn't depend on the specific phrase extraction algorithm used initially.
Datasets
- Training Data: BTEC corpus (162,318 lines of parallel Japanese-English text).
- Language Model Training: English portion of the BTEC corpus.
- Test Data: IWSLT 2004 tourism data (500 lines) with 16 reference translations.
The authors also mention using the Japanese training data (162,318 lines) to collect the usage statistics for pruning.
Key Insights and Ideas
- Usage is Key: The most important insight is that simply having a high probability or being considered frequently doesn't guarantee a phrase pair is useful. Actual usage in the final translation is a much better indicator of its value.
- Pruning for Portability: The motivation for pruning is directly tied to the growing need for SMT on memory-constrained devices like mobile phones, making the research highly relevant to practical applications.
- Nuanced Pruning: This method offers a more sophisticated way to prune than simple thresholds, leading to better trade-offs between model size and translation quality.
- Infrequent Source Phrases Targeted: The improved pruning method tends to remove infrequent source phrases, which is a sensible heuristic for reducing redundancy.
Key Results
The proposed pruning method demonstrated significant improvements over baseline pruning techniques.
- High Pruning Rate: Over 50% of phrase pairs could be pruned without a significant loss of translation performance.
- Performance Gains: At a model size of 200,000 phrase pairs, the proposed method achieved a relative improvement of up to 58% compared to baseline pruning.
- Comparable Performance: Even with millions of phrase pairs removed, the proposed method's translation score (BLEU) remained comparable to the baseline, with the baseline dropping faster as more pairs were removed. For example, at 2 million phrase pairs, the proposed method's score was still close to the best possible baseline score.
- Figure 1 visually shows that the pruned model's BLEU score closely tracks the baseline score, but allows for a much smaller number of phrase pairs.
Limitations and Future Work
- Computing Power Not Addressed: The paper focuses solely on memory limitations and does not address the computational power constraints of portable devices.
- Metric-Best Path vs. Model-Best Path: The authors suggest exploring whether using a "metric-best path" (requiring reference translations) instead of the "model-best path" for statistics collection would yield better results, potentially with smaller datasets.
- Sequential Pruning: They propose investigating a sequential pruning approach, similar to Optimal Brain Damage, where statistics are recalculated after removing each set of phrase pairs, rather than a one-off calculation.
Summary updated