A Unigram Orientation Model for Statistical Machine Translation

Christoph TillmannNAACL 2004

TL;DR

This paper introduces a novel "unigram orientation model" for statistical machine translation (SMT) that improves translation quality by explicitly modeling the relative positioning of phrase pairs (blocks). By learning how often blocks appear to the left or right of their predecessors, the model effectively handles word reordering, a key challenge in SMT.

What Problem Does This Paper Address?

Statistical machine translation systems often struggle with reordering words and phrases between languages, especially when the target language's word order differs significantly from the source. Traditional phrase-based SMT systems typically rely on a "monotone" translation process, where phrases are translated in the same order they appear in the source. While some systems allow for reordering, controlling this process effectively without introducing errors is difficult. This paper tackles the problem of improving translation quality by introducing a more nuanced way to handle reordering, specifically focusing on the relative orientation of adjacent phrase blocks.

Methods

The core contribution is the unigram orientation model. This model extends a standard phrase-based SMT system by treating translation as a block segmentation process. Instead of just considering the probability of a block appearing, it also considers its orientation relative to the preceding block.

Here's a breakdown of the key techniques:

  • Block Segmentation: The input sentence is segmented into "blocks," which are essentially pairs of phrases without internal structure.
  • Unigram Orientation Model: During training, the system counts how often a block appears to the left (NL(b)) or right (NR(b)) of its predecessor. This is a "unigram" model because it looks at individual block orientations, not complex multi-block relationships.
  • Orientation Component: This learned orientation probability (P_b(o)) is then incorporated into the overall block bigram score.
  • Log-Linear Combination: The final translation score is a log-linear combination of unigram block probabilities, block bigram probabilities, and the new orientation probabilities.
  • Decoding: A beam search procedure is used to find the best block sequence with optimal orientation. Crucially, the model restricts reordering to swapping neighboring blocks, preventing drastic reordering that could lead to errors.

What's new is the explicit modeling of the orientation of a block relative to its predecessor, using simple unigram counts. This is distinct from more complex distortion models that might require full block alignment.

Datasets

  • Training Data: UN news sources, consisting of 87.5 million Arabic words and 97.1 million English words, yielding 3.3 million sentence pairs. The Arabic data is romanized.
  • Devtest Set: LDC data, comprising 1043 sentences with 25889 Arabic words and 4 reference translations. This set is used for tuning parameters.
  • Blind Test Set: MT 03 Arabic-English DARPA evaluation test set, with 663 sentences and 16278 Arabic words.

Key Insights and Ideas

  • Orientation Matters: Explicitly modeling the relative orientation of adjacent phrase blocks significantly improves translation quality. This suggests that word reordering isn't just about arbitrary swaps, but has a structured, local component.
  • Neighbor Swapping is Key: Restricting reordering to only adjacent block swaps is a practical and effective way to manage reordering without overwhelming the system with too many possibilities.
  • Unigram Counts are Sufficient: Surprisingly, simple unigram counts of block orientations are enough to capture useful reordering information, avoiding the need for more complex alignment models.
  • Language Model Alone is Weak for Reordering: A standard trigram language model, when used alone to control block swapping, is insufficient and can lead to worse performance, highlighting the need for a dedicated reordering model.

Key Results

The paper presents three systems:

  • S0: Baseline unigram model without reordering (monotone).
  • S1: Block swapping model, but reordering is controlled only by a trigram language model.
  • S1 & OR: The proposed model, incorporating the unigram orientation component.

On both the LDC devtest set and the DARPA MT 03 blind test set, the S1 & OR model achieved statistically significant improvements in BLEU scores over the baseline S0 model. For instance, on the DARPA MT 03 test set, S1 & OR achieved a BLEU score of 0.356 ± 0.017, compared to S0's 0.339 ± 0.016. Interestingly, the S1 model (swapping without orientation) performed worse than the baseline S0, underscoring the importance of the orientation model.

Limitations and Future Work

  • Normalization: The normalization over successor blocks in the scoring equations is computationally infeasible due to the large number of possible blocks. The authors note that this might be less critical for their segmentation model as it naturally favors fewer, larger blocks.
  • Predecessor Orientation Ignored: The current orientation model ignores the orientation of the predecessor block (o'). The authors suggest that future work could incorporate this, potentially leading to even better control over swapping.
  • Restricted Successor Set: For future work, they propose normalizing over a restricted successor set, such as blocks that match a given source input sentence, to improve efficiency.
  • Position-Dependent Distortion: The current approach doesn't capture position-dependent distortion probabilities, which could be an area for future exploration.

Summary updated

BibTeX
@inproceedings{tillmann2004unigram,
  author = {Christoph Tillmann},
  title = {{A Unigram Orientation Model for Statistical Machine Translation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2004},
}
Figure