TL;DR
This paper introduces a novel approach to estimating phrase translation probabilities in statistical machine translation (SMT) by leveraging the Dempster-Shafer theory of belief functions. By modeling uncertainty more flexibly than traditional probability methods, this new technique shows comparable or improved translation quality, particularly when combined with existing methods.
What Problem Does This Paper Address?
Statistical Machine Translation (SMT) relies heavily on phrase tables, which store potential translations for source language phrases along with their probabilities. Estimating these probabilities accurately is crucial for good translation quality. The classical approach uses relative frequencies from large parallel corpora (bitexts). However, this method has limitations, especially with rare or ambiguous phrase pairs. For instance, a unique phrase pair might get a probability of 1, even if it's not a reliable translation, simply because it appears only once. This can lead to issues with uncertainty and conflict between translation hypotheses, which traditional probability theory struggles to handle effectively.
Methods
The core innovation of this paper is the application of the Transferable Belief Model (TBM), a framework within the Dempster-Shafer theory, to estimate phrase translation features. Instead of directly assigning probabilities, TBM assigns "belief masses" to propositions.
Here's how it works:
- Information Representation: For a given source phrase, each potential target phrase translation is treated as a piece of information. This information is represented by a belief function
m, wherem({t})is the probability of that specific translationtgiven the sources, andm(T_s)represents the remaining belief (uncertainty or ignorance) about the translation ofs. - Combining Information: The TBM allows for the combination of multiple belief functions using a conjunctive operator (denoted by
∩). This operator is designed to handle conflicts between different pieces of information. In this paper, they combine the belief functions derived from all possible target translations for a given source phrase. - Feature Estimation: The resulting combined belief mass for a specific target phrase
t_iis then used as the feature valuef(t_i, s_j). This replaces the traditional relative frequency estimation. The same process is applied to estimate the inverse translation probabilities (target to source).
Datasets
The experiments were conducted on several datasets:
- WMT task: This involved translation between French and English, using the
nwtst2010(development) andnwtst2011(test) corpora. - COSMAT corpus: This dataset focuses on translating abstracts of PhD theses in computer science between French and English.
- Arabic-English translation: This used data from the GALE program, specifically the
nist09-nw(development) andnist08-nw(test) corpora for news translation.
Key Insights and Ideas
- Beyond Probabilities: The paper highlights that belief functions offer a more nuanced way to represent uncertainty and handle conflicts compared to standard probability distributions, which is particularly beneficial in the noisy domain of machine translation.
- Handling Rare Events: The TBM approach shows promise in better estimating probabilities for rare phrase pairs, where traditional methods can be unreliable.
- Synergy with Classical Methods: The most significant finding is that combining the belief function approach with the classical probability-based estimation often yields the best results, suggesting a complementary relationship between the two.
Key Results
The paper reports that the belief function approach, when applied to phrase translation features, achieves results that are at least comparable to, and sometimes better than, the classical relative frequency method.
- WMT (English-French): The belief function approach showed a slight improvement in BLEU score.
- Arabic-English: While some experiments showed a slight decrease on development sets, improvements were observed on test sets.
- Combined Approach: Combining the belief function approach with the classical method consistently showed improvements across various tasks and language pairs, with BLEU score increases of up to 0.2-0.3 points and corresponding improvements in TER scores. For instance, on the WMT task, the combined approach improved BLEU by about 0.2 points on the tuning corpus and 0.15 on the test corpus for English-to-French translation.
Limitations and Future Work
- Focus on Phrase Translation: The current work only applies the belief function approach to phrase translation features, leaving other features (like lexical weighting) to their classical estimation.
- Optimization Sensitivity: Some experiments showed high standard deviations with MERT optimization, though MIRA optimization generally yielded more stable and precise results.
- Future Directions: The authors suggest applying this approach as a secondary phase-table for rescoring hypotheses during the decoding process, or on n-best translation outputs. This could further enhance translation quality by leveraging the belief function's ability to handle uncertainty and conflicts in a more sophisticated manner.
Summary updated