Comparison of Extended Lexicon Models in Search and Rescoring for SMT

Saša Hasan, Hermann NeyNAACL 2009

TL;DR

This paper introduces an "extended lexicon model" for statistical machine translation (SMT) that uses two source words to predict a target word, capturing longer-range dependencies than traditional models. They demonstrate that integrating this model directly into the search process, rather than just in a post-hoc rescoring step, leads to significant improvements in translation quality.

What Problem Does This Paper Address?

Statistical Machine Translation (SMT) systems, particularly phrase-based ones, have seen tremendous progress. However, a persistent challenge is effectively incorporating broader contextual information beyond local word or phrase alignments. Standard SMT models often rely on n-gram language models and bilingual phrase tables, which inherently have a limited view of the sentence. This can lead to suboptimal translation choices when the correct translation of a word or phrase depends on distant words in the source sentence. The authors aim to address this by developing a model that can leverage more global source context to inform translation decisions.

Methods

The core innovation is the extended lexicon model, which moves beyond the standard $p(e|f)$ (probability of target word $e$ given source word $f$) to $p(e|f, f')$. Here, $f'$ is a "trigger" word from the source sentence. This allows the model to make more nuanced translation choices based on not just the directly aligned source word, but also another related source word, potentially far away.

To manage the complexity of this model, they employ a path-constrained triplet model. This means the first source word ($f$) is constrained to be aligned to the target word ($e$), while the second trigger word ($f'$) can be any word in the source sentence. This helps keep the number of triplets manageable.

They compare two integration strategies:

  1. Rescoring: Applying the extended lexicon model to re-rank a set of candidate translations (n-best lists) generated by a baseline system.
  2. Search Integration: Incorporating the extended lexicon model directly into the decoder's search process, influencing phrase pair scoring during translation generation.

Datasets

The experiments were conducted on the GALE Chinese-English task.

  • Training Data: 32.3 million running words on the English side, with a vocabulary of 76.5K source and 241.7K target words. The extended lexicon model was trained on a subset of this data, resulting in approximately 62 million triplets.
  • Development Data: Used for tuning feature weights.
  • Test Sets:
    • Newswire: Evaluated on the GALE 2008 test set.
  • Web Text: Evaluated on the GALE 2008 test set.

Key Insights and Ideas

  • Context is King (Even Distant Context): The extended lexicon model's ability to use a second source word trigger significantly improves translation quality, suggesting that long-distance dependencies are crucial for accurate translation.
  • Search Integration Outperforms Rescoring: Integrating the model directly into the search process yields better results than applying it only during rescoring. This is likely because search integration influences the generation of hypotheses, allowing for better initial translations, whereas rescoring only re-orders existing ones.
  • Combining Approaches is Best: The highest gains were achieved by using the extended lexicon model in search and then applying it again in rescoring (using the reverse direction $p(f|e, e')$ which is only feasible in rescoring).
  • IBM Model 1 as a Baseline: The paper notes that their extended model builds upon the standard IBM Model 1, suggesting a clear lineage and incremental improvement.

Key Results

  • Newswire:
    • Baseline: 32.5% BLEU / 59.4% TER.
  • Rescoring with additional models (no triplets): Slight improvement.
  • Adding triplet model in search ($p(e|f, f')$): +0.6% BLEU.
  • Adding triplet model in search and rescoring (both directions): +0.9% BLEU / -1.5% TER absolute improvement over the rescoring baseline without triplets. This is a significant gain.
  • Web Text:
    • Baseline: 25.8% BLEU / 64.0% TER.
  • Adding triplets to search: Only a small improvement (+0.2% BLEU). The authors hypothesize this is because the triplet lexicon was trained primarily on newswire data.
  • Rescoring without triplets (using Google Web LM) showed a large improvement.

Limitations and Future Work

  • Data Sparsity: The authors mention smoothing the model with a floor value of $10^{-7}$ to handle unseen events, indicating data sparsity is a concern.
  • Web Text Performance: The triplet model did not perform as well on web text, suggesting the need for training data that better reflects the target domain or exploring domain adaptation techniques.
  • Reverse Direction in Search: The $p(f|e, e')$ direction of the triplet model is difficult to use directly in search because the full target hypothesis isn't available. This limits its application to rescoring.
  • Future Work:
    • Investigate more variants of the triplet model.
  • Apply the models to other language pairs (e.g., French-English, German-English).
  • Explore cross-lingual trigger models or constrained variants.
  • Find ways to incorporate the reverse direction ($p(f|e, e')$) more effectively into the search process.

Summary updated

BibTeX
@inproceedings{hasan2009comparison,
  author = {Saša Hasan and Hermann Ney},
  title = {{Comparison of Extended Lexicon Models in Search and Rescoring for SMT}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2009},
}
Figure