Simultaneous Word-Morpheme Alignment for Statistical Machine Translation

Elif Eyigöz, Daniel Gildea, Kemal OflazerNAACL 2013

TL;DR

This paper introduces a novel two-level alignment model that simultaneously aligns words and their constituent morphemes for statistical machine translation (SMT). By explicitly modeling morphology, their approach significantly improves translation quality, particularly for morphologically rich languages like Turkish, outperforming traditional word-alignment models.

What Problem Does This Paper Address?

Statistical machine translation (SMT) systems heavily rely on word alignments derived from parallel corpora. However, existing word alignment models typically treat words as atomic units, ignoring their internal morphological structure. This simplification is a significant drawback when dealing with morphologically rich languages, where a single word can carry a lot of grammatical and semantic information through its morphemes. Researchers have previously attempted to handle this by simply splitting words or using techniques like lemmatization, but these methods often fail to capture the full complexity of morphological relationships. This paper addresses this gap by proposing a model that explicitly aligns words and their morphemes, aiming to leverage morphological information for better translation.

Methods

The core contribution is a Two-level Alignment Model (TAM). TAM operates on two levels:

  1. Word Alignment: This level is analogous to standard IBM Model 1, aligning words between source and target sentences.
  2. Morpheme Alignment: This level operates within the scope of a given word alignment. It aligns the morphemes of a source word to the morphemes of its target word.

TAM jointly learns both word and morpheme alignments using an Expectation-Maximization (EM) algorithm. The model is developed in two stages:

  • Initial Model: A "bag-of-words" and "bag-of-morphemes" approach, essentially embedding one IBM Model 1 within another.
  • HMM Extension: This stage introduces distortion probabilities at the word level, building upon the initial model to incorporate sequential dependencies, similar to Hidden Markov Models (HMMs).

The paper also discusses two versions of TAM:

  • Word-and-Morpheme: The full model.
  • Morpheme-only: A simplified version that focuses solely on morpheme alignments, which is shown to be "non-deficient" (less prone to probability allocation issues).

To handle rare words and prevent overfitting, the authors employ a Variational Bayes extension of the EM algorithm.

Datasets

  • Training and Evaluation: A Turkish-English parallel corpus of approximately 50,000 sentences.
  • Morphological Analysis:
    • Turkish data was morphologically parsed and disambiguated.
  • English data was segmented using the CELEX database.
  • Gold Standard Alignments: 75 Turkish-English sentences were manually aligned to serve as a gold standard for evaluating word alignment quality.

Key Insights and Ideas

  • Morphology Matters for Alignment: Explicitly modeling morphemes significantly improves word alignment quality, leading to better translation performance.
  • Two-Level Structure is Key: The hierarchical structure of aligning words first, then their morphemes within that word alignment, provides a principled way to integrate morphological information.
  • Joint Learning is Effective: Simultaneously learning word and morpheme alignments, rather than treating them as separate steps, allows the model to discover complex interdependencies.
  • Morpheme-only Model's Robustness: The "morpheme-only" version of TAM, while simpler, performs surprisingly well and is less susceptible to certain modeling deficiencies.
  • Dirichlet Priors' Nuanced Impact: Dirichlet priors, used for anti-smoothing, benefit the word-only model more than TAM models, likely because word vocabularies are larger and have more rare words than morpheme vocabularies.

Key Results

  • Significant BLEU Score Improvement: TAM models achieved significant improvements in BLEU scores over IBM Model 4 on Turkish-English translation tasks.
  • Reduced Alignment Error Rate (AER): The proposed models also showed lower AER compared to baseline word-alignment models.
  • Word-and-Morpheme vs. Morpheme-only: In four out of eight experimental settings, the morpheme-only model outperformed the word-and-morpheme model, suggesting the importance of segmentation accuracy and language's morphological richness.
  • Comparison to IBM Model 4: TAM models consistently outperformed IBM Model 4 across various settings.

Limitations and Future Work

  • Deficiency in Word-and-Morpheme Model: The authors acknowledge that the word-and-morpheme version of TAM is "deficient" because it can allocate probability to cases where morpheme translations don't match word translations, and single-morpheme words are generated twice (as words and morphemes). However, they note that this deficiency didn't always lead to worse performance.
  • HMM Extension Scope: The HMM extension was applied only at the word level. The paper suggests that an HMM extension for the morpheme-only version is also possible.
  • Segmentation Dependency: The superiority of the morpheme-only model is noted to be highly dependent on segmentation accuracy and the morphological richness of the language.
  • Future Work: The paper implies that further exploration of the morpheme-only model and its HMM extension, as well as investigating the impact of segmentation quality, would be valuable.

Summary updated

BibTeX
@inproceedings{eyigoz2013simultaneous,
  author = {Elif Eyigöz and Daniel Gildea and Kemal Oflazer},
  title = {{Simultaneous Word-Morpheme Alignment for Statistical Machine Translation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2013},
}
Figure