TL;DR
This paper introduces a novel approach to improve machine translation (MT) quality by using "word confidence" scores to re-rank the top-N translation hypotheses. By integrating these confidence scores as new features into a standard re-ranking model, the authors demonstrate a significant improvement in translation quality, especially when simulating a near-perfect confidence estimation system.
What Problem Does This Paper Address?
Machine translation systems often produce multiple candidate translations (an "N-best list") for a given source sentence. The system typically selects the "best" translation based on an internal scoring function. However, this best-ranked translation isn't always the one that a human would prefer. This gap between the system's best guess and human judgment is a persistent challenge. Traditional methods to improve translations after decoding include post-editing or re-decoding. This paper focuses on re-ranking, a technique that uses additional features beyond the decoder's original scores to re-evaluate and re-order the N-best list, aiming to bring a more human-like translation to the top. The specific problem addressed is how to effectively leverage "word confidence" information – an estimate of how likely each word in a translation is correct – to improve this re-ranking process.
Methods
The core of the proposed method is to extract six new features from Word Confidence Estimation (WCE) scores and integrate them into a log-linear model for re-ranking.
- Word Confidence Estimation (WCE): The paper uses a Conditional Random Fields (CRFs) model, trained with the WAPITI toolkit, to predict word confidence. The WCE system is trained on a rich set of features, including target and source word information, alignment context, word posterior probabilities, pseudo-references, graph topology, language model features, and various lexical, syntactic, and semantic features.
- Feature Engineering for Re-ranking: The word-level confidence labels from the WCE system are synthesized into sentence-level scores. These six proposed scores are:
- Ratio of "good" words to total words.
- Ratio of "good" nouns/verbs to total nouns/verbs.
- Ratios of consecutive "good" word sequences (bigrams, trigrams, and quadrigrams).
- Log-linear Re-ranking Model: These six WCE-based scores are combined with the existing 14 decoder scores in a log-linear model. The weights of all 20 scores are then optimized using either Minimum Error Rate Training (MERT) or Margin Infused Relaxed Algorithm (MIRA).
Datasets
The experiments were conducted on a French-English dataset.
- Training Data: 10,000 French sentences with their corresponding 1000-best MT hypotheses and human post-editions were used to train the WCE model.
- Test Data: 881 French sentences were used for evaluation. For these sentences, 1000-best lists were generated, and WCE features were extracted for all hypotheses.
- Oracle WCE: To simulate an ideal WCE system, the authors used oracle labels derived from human post-editions. This "oracle" scenario represents a hypothetical WCE system that perfectly identifies correct and incorrect words.
Key Insights and Ideas
- Correlation is Key: The paper first establishes a strong correlation between sentence-level WCE scores (specifically, the percentage of "good" words) and standard MT evaluation metrics like BLEU, TER, and TERp-A. This provides a strong theoretical basis for using WCE in re-ranking.
- WCE as a Complementary Signal: WCE scores, derived independently from the decoder, offer a valuable signal that complements the existing decoder features. They capture aspects of translation quality that the decoder might miss.
- The Power of "Oracle": The experiments with a simulated "oracle" WCE system show a dramatic improvement in MT quality. This highlights the immense potential of WCE if it can achieve near-perfect accuracy.
- Gradual Improvement: Simulating WCE systems with increasing accuracy (25%, 50%, 75% correction of errors) demonstrates a clear, linear improvement in MT quality. This suggests that even imperfect WCE systems can offer benefits, and better WCE directly translates to better MT.
Key Results
- Modest Improvement with Real WCE: The real WCE system, which achieved first rank in the WMT 2013 Quality Estimation Task, provided a modest but statistically significant improvement in BLEU scores (around 0.46 points with MERT optimization) over the baseline system. TER and TERp-A improvements were negligible.
- Significant Boost with Oracle WCE: The "oracle" WCE system achieved substantial gains. For example, it boosted BLEU scores by 5.79 points (MERT) and 4.72 points (MIRA) compared to the baseline. The oracle system's performance reached 60.48 BLEU, which is very close to the theoretical maximum.
- Simulated WCE Performance: Simulated WCE systems with 25%, 50%, and 75% error correction showed increasing improvements in BLEU scores (0.68, 3.00, and 3.63 points respectively with MERT).
- MERT vs. MIRA: For the tested configurations, MERT optimization generally yielded better results than MIRA.
Limitations and Future Work
- Limitations of Real WCE: The paper acknowledges that the real WCE system's performance is limited, particularly in predicting "bad" words. This explains why the improvements with the real WCE system are modest.
- Future Work:
- Augmenting WCE Features: The authors plan to improve WCE performance by incorporating more linguistic features and exploring advanced techniques like feature selection and boosting.
- Integrating WCE into the Decoder: A promising future direction is to integrate WCE scores directly into the decoder's search graph to guide the decoding process itself, rather than just re-ranking. Preliminary experiments in this area are noted as promising.
Summary updated