A Sense-Based Translation Model for Statistical Machine Translation

Deyi Xiong, Min ZhangACL 2014 · DOI · OpenReview

TL;DR

This paper introduces a novel sense-based translation model for Statistical Machine Translation (SMT) that directly incorporates automatically learned word senses from the source language. By using a nonparametric Bayesian topic model for word sense induction and a maximum entropy classifier for translation, they demonstrate significant improvements over baseline SMT and previous reformulated word sense disambiguation approaches.

What Problem Does This Paper Address?

Statistical Machine Translation (SMT) systems often struggle with word ambiguity, where a single word can have multiple meanings, leading to incorrect translations. While word sense disambiguation (WSD) has been explored to address this, prior attempts either reformulated WSD to predict target translations directly (sidestepping explicit word senses) or used predefined sense inventories, which are often insufficient for large-scale, open-domain translation. This paper tackles the question of whether directly integrating automatically learned word senses can genuinely improve SMT quality, moving beyond previous limitations.

Methods

The core of this work is a two-pronged approach:

  1. Word Sense Induction (WSI) for Sense Tagging: The authors employ a Hierarchical Dirichlet Process (HDP)-based topic model for WSI. Unlike traditional WSD, this method automatically learns sense clusters for words without relying on a predefined lexicon. For each source word type, they create "pseudo-documents" from its surrounding contexts and use the HDP to infer latent senses (topics). This allows for a broad-coverage sense tagger that can annotate source words in training, development, and test sets.
  2. Sense-Based Translation Model: They propose a maximum entropy (MaxEnt) classifier to integrate these learned word senses into the SMT decoder. This model predicts the probability of translating a source word into a target phrase, considering not only standard lexicon features (neighboring words) but also sense features derived from the HDP-based WSI. This differs from previous WSD-for-SMT approaches by explicitly using the induced word senses as features.

Datasets

The experiments were conducted on a large-scale Chinese-to-English translation task using LDC corpora, specifically LDC2004E12, LDC2004T08, LDC2005T10, LDC2003E14, LDC2002E18, LDC2005T06, LDC2003E07, and LDC2004T07. This dataset comprises approximately 3.84 million bilingual sentences. The NIST MT03 dataset was used as the development set, and NIST MT05 as the test set. No new benchmarks were introduced; rather, existing large-scale datasets were utilized.

Key Insights and Ideas

  • Word Senses are Directly Useful for SMT: The paper provides empirical evidence that automatically learned word senses, when directly integrated, can significantly improve SMT quality. This counters earlier findings where WSD approaches sometimes degraded performance.
  • HDP for Broad-Coverage WSI: Using a nonparametric Bayesian model like HDP is crucial for learning a large number of senses without pre-specification, making it suitable for large-scale, open-domain translation.
  • Sense Features Complement Lexicon Features: The sense features derived from WSI provide distinct semantic information that complements traditional lexicon features, leading to better translation predictions.
  • "Micro-Topics" for Translation: The induced word senses can be viewed as "micro-topics" specific to individual words, which are more granular and directly applicable to translation than generic document-level topics.

Key Results

The proposed sense-based translation model (STM) demonstrated substantial improvements:

  • Overall Improvement: The STM achieved a 1.2 BLEU point improvement over the baseline SMT system.
  • Sense Features Alone: Even when using only sense features (without lexicon features), the model still outperformed the baseline by 0.62 BLEU points, highlighting the inherent value of induced word senses.
  • Comparison to Reformulated WSD: The STM outperformed the previous reformulated WSD approach by 0.57 BLEU points, indicating that directly using learned word senses is more effective than simply predicting target translations.
  • Window Size: The optimal context window size for extracting features for the MaxEnt classifier was found to be ±10 words.

Limitations and Future Work

  • Limitations:
    • The HDP-based WSI models are not trained for extremely frequent words (more than 20,000 occurrences) to expedite training.
  • Unseen words in the test/dev set are assigned a default sense.
  • The current model focuses on source-side word senses.
  • Future Work:
    • Explore both "micro-topics" (word senses) and "macro-topics" (document topics) for machine translation.
  • Induce sense clusters for words in the target language to build sense-based language models.
  • Investigate whether automatically learned senses of preceding words can aid in predicting succeeding words.

Summary updated

Abstract

The sense in which a word is used determines the translation of the word. In this paper, we propose a sense-based translation model to integrate word senses into statistical machine translation. We build a broad-coverage sense tagger based on a nonparametric Bayesian topic model that automatically learns sense clusters for words in the source language. The proposed sense-based translation model enables the decoder to select appropriate translations for source words according to the inferred senses for these words using maximum entropy classifiers. Our method is significantly different from previous word sense disambiguation reformulatedfor machine translation in that the latter neglects word senses in nature. We test the effectiveness of the proposed sensebased translation model on a large-scale Chinese-to-English translation task. Results show that the proposed model substantially outperforms not only the baseline but also the previous reformulated word sense disambiguation.
BibTeX
@inproceedings{xiong2014sense,
  author = {Deyi Xiong and Min Zhang},
  title = {{A Sense-Based Translation Model for Statistical Machine Translation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2014},
  doi = {10.3115/v1/p14-1137},
}
Figure