TL;DR
This paper systematically compares various training criteria for statistical machine translation (SMT) systems, introducing novel methods based on maximum likelihood estimation and expected loss. They demonstrate that optimizing for expected loss, particularly using the Minimum Bayes Risk (MBR) decision rule, significantly improves translation quality over the standard Minimum Error Rate Training (MERT) approach, especially on large-scale tasks.
What Problem Does This Paper Address?
Statistical machine translation systems rely on numerous parameters that need to be "trained" from data. This training process aims to optimize the system's performance according to a chosen evaluation metric. The authors identify that while the Bayes decision rule for SMT is well-defined, the "training problem" – how to estimate these free parameters – is crucial. Existing methods, like MERT, optimize directly for an evaluation criterion but can be sensitive to specific choices. This paper seeks to explore and compare a wider range of training criteria, including new ones, and evaluate their effectiveness, particularly in conjunction with different decision rules (MAP vs. MBR).
Methods
The paper introduces and compares several training criteria:
- Maximum Likelihood Estimation (MLE):
- Sentence-Level MLE: Aims to maximize the likelihood of the correct translation for each sentence. A novel variant uses the translation with the maximum BLEU score as a "pseudo-reference" to bias the system towards BLEU.
- N-gram Level MLE: A more fine-grained approach that assumes conditional independence among n-grams. This method uses human reference translations and accounts for partial correctness.
- Expected Loss Computation:
- Expected BLEU Score: This is a theoretically sound approach based on statistical decision theory, aiming to maximize the expected BLEU score. The paper proposes two variants:
- Sentence-Level Expected BLEU: Computes the expected BLEU over all possible translations, approximated using an N-best list.
- N-gram Level Expected BLEU: Exploits the structure of the BLEU score to derive a more efficient computation, considering expected brevity penalty and n-gram precisions.
The paper also contrasts two decision rules:
- Maximum A-Posteriori (MAP): Selects the hypothesis with the highest posterior probability. This is computationally simpler and often used.
- Minimum Bayes Risk (MBR): Selects the hypothesis that minimizes the expected loss (or maximizes expected gain) over all possible translations. This is theoretically optimal but computationally more demanding.
Datasets
The experiments were conducted on the Chinese-English TC-Star task. This is a broadcast news speech translation task. The training data consists of publicly available LDC Chinese-English corpora. The evaluation was performed on the official evaluation test sets for the years 2006 and 2007. The development set and test sets each had two reference translations.
Key Insights and Ideas
- MBR Outperforms MAP: The Minimum Bayes Risk (MBR) decision rule consistently yields better translation quality (measured by BLEU score) than the Maximum A-Posteriori (MAP) rule, especially when combined with more sophisticated training criteria. This suggests that optimizing for expected loss is more robust than simply picking the most probable translation.
- Expected BLEU is Powerful: Training criteria based on maximizing the expected BLEU score (both sentence-level and n-gram level) achieve the best results, outperforming even the state-of-the-art MERT baseline. This validates the theoretical soundness of optimizing for expected evaluation metrics.
- N-gram Level MLE is Better than Sentence-Level: The n-gram level MLE criterion performs better than the sentence-level MLE, likely because it doesn't require pseudo-references and can account for partial correctness.
- MBR's Robustness: The MBR decision rule appears to be less affected by sub-optimal parameter settings, suggesting better generalization capabilities.
- Pseudo-References are Problematic: Using pseudo-references (like the highest BLEU candidate) in MLE can bias the system but doesn't guarantee optimality with respect to the BLEU score itself.
Key Results
- The proposed Expected BLEU score training criteria achieved the best results, improving the BLEU score by 1.0-1.8% absolute over the MERT baseline on the test sets, depending on the decision rule and test year.
- The MBR decision rule consistently outperformed the MAP decision rule across most training criteria. For example, on the Eval'06 set, MBR achieved higher BLEU scores than MAP for most criteria.
- The MERT-Bleu baseline achieved a BLEU score of 19.5% on the development set.
- On the development data, MBR decoding compensated for weaker training criteria, leading to similar BLEU scores (around 18.9%) for various criteria. However, the benefits of MBR became more pronounced on the test sets.
Limitations and Future Work
- Computational Cost of MBR: The paper notes that computing MBR hypotheses is more time-consuming than MAP. The authors suggest that a more efficient algorithm for computing MBR hypotheses would be desirable.
- N-gram Level Expected BLEU: While the n-gram level expected BLEU computation yielded good results, the authors suggest that using word graphs for computing n-gram posterior probabilities could lead to more reliable estimates, which is an avenue for future work.
- Statistical Significance: While improvements are reported, the paper mentions that in rare cases where MAP yielded better results, the differences were small and not statistically significant.
- Generalizability of Expected BLEU: The paper focuses on BLEU, but the methods for computing expected BLEU are presented as adaptable to other n-gram based metrics, suggesting future work in exploring these.
Summary updated