Improved Reordering for Phrase-Based Translation using Sparse Features

Colin CherryNAACL 2013 · OpenReview

Abstract

Recent advancements in Statistical Machine Translation (SMT) have explored tuning systems with large numbers of sparse features. However, practical examples of beneficial sparse features for phrase-based systems remain limited. This paper addresses reordering—a traditional weakness of phrase-based translation—by introducing sparse features into a hierarchical reordering model (HRM). By coupling phrase orientation with frequent words or word clusters, we demonstrate significant improvements in translation quality, achieving gains of up to 1.2 BLEU points in Chinese-English and 1.8 BLEU points in Arabic-English tasks. Our results indicate that sparse decoder features, optimized directly for BLEU, significantly outperform traditional maximum entropy models trained on word-aligned bitext.

Introduction: The Reordering Challenge

In Phrase-Based Statistical Machine Translation (SMT), "reordering"—the process of determining the correct word order in the target language—is a persistent challenge. Standard models typically rely on "lexicalized reordering," which calculates the probability of orientations (e.g., Monotone, Swap, or Discontinuous) based on their relative frequencies in a training corpus.

A major hurdle is data sparsity. Most phrase pairs appear infrequently, leaving the model with insufficient statistics to handle new or rare phrases. While researchers have attempted to mitigate this using Maximum Entropy (MaxEnt) models to "smooth" these statistics across millions of sentences, these models often fail to deliver proportional gains in actual translation quality. This is primarily due to an "objective mismatch": MaxEnt models optimize for predicting training data (likelihood), which does not always correlate with producing better translations (BLEU).

Methodology

This research compares three distinct approaches to the Hierarchical Reordering Model (HRM):

  1. Standard HRM (Baseline): Employs relative frequencies from word-aligned bitext, smoothed using MAP (Maximum A Posteriori) techniques.
  2. Maximum Entropy HRM: A complex model trained on millions of bitext examples. It uses features like the first and last words of a phrase and word clusters to predict reordering orientations.
  3. Sparse Feature HRM: The proposed approach integrates thousands of simple indicator features directly into the translation decoder. These features (e.g., "Does this phrase start with word X and result in a Swap?") are tuned on a small development set (a few thousand sentences) using an optimizer like MIRA to directly maximize the BLEU score.
The sparse features are constructed using templates that combine phrase boundaries with word representations and orientations.

The sparse features utilize three levels of representation: lexical (actual words), 50-word clusters, and 20-word clusters. This hierarchy allows the model to generalize from specific lexical items to broader grammatical categories.

Experimental Setup

Experiments were conducted on two large-scale translation tasks:

  • Arabic-to-English: Trained on 1.49 million sentence pairs.
  • Chinese-to-English: Trained on 3.5 million sentence pairs. Evaluation utilized NIST MT evaluation data, with benchmarks including mt06, mt08, and mt09.

Key Insights

The "Decoder in the Loop" Advantage

The primary finding is that optimizing directly for translation quality (BLEU) on a small, representative dataset is more effective than optimizing for likelihood on a massive one. Sparse features allow the optimizer to observe the decoder's actual behavior and correct specific errors in situ.

Objective Mismatch

The study highlights a disconnect between model perplexity and translation performance. As shown in the comparison between MaxEnt and relative frequency models, increasing training data significantly improves a model's ability to predict alignments (lower perplexity), but these improvements do not consistently translate into higher BLEU scores.

Comparison of BLEU and Perplexity as training data grows. While MaxEnt achieves lower perplexity, its BLEU advantage over the baseline diminishes as more data is added.

Simplicity and Context

Effective reordering does not require deep linguistic or syntactic parsing. Simply tracking the "corners" of a phrase—the first and last words or clusters—provides a sufficient signal for the optimizer. Furthermore, these features are most effective when they have access to the decoder's "stack" context, allowing them to account for the preceding translated content.

Results and Performance

  • Significant BLEU Gains: The Sparse HRM outperformed the baseline by 1.2 BLEU points in Chinese-English and 1.8 BLEU points in Arabic-English.
  • Superiority over MaxEnt: In direct comparisons, sparse features tuned for BLEU consistently outperformed MaxEnt models, even those trained on significantly more data.
  • Robustness: The approach provided a consistent boost (0.4 to 1.1 BLEU) even when integrated into high-performance SMT systems already utilizing thousands of other features.

Conclusion and Future Directions

While the Sparse HRM offers substantial quality improvements, it introduces a computational overhead, running approximately four times slower than the baseline decoder in experimental settings. Future work may focus on pre-calculating weights to mitigate this latency. Additionally, exploring "transition-based" features that look deeper into the reordering stack could further enhance long-distance reordering capabilities.

Summary updated

Abstract

There have been many recent investigations into methods to tune SMT systems using large numbers of sparse features. However, there have not been nearly so many examples of helpful sparse features, especially for phrasebased systems. We use sparse features to address reordering, which is often considered a weak point of phrase-based translation. Using a hierarchical reordering model as our baseline, we show that simple features coupling phrase orientation to frequent words or wordclusters can improve translation quality, with boosts of up to 1.2 BLEU points in ChineseEnglish and 1.8 in Arabic-English. We compare this solution to a more traditional maximum entropy approach, where a probability model with similar features is trained on wordaligned bitext. We show that sparse decoder features outperform maximum entropy handily, indicating that there are major advantages to optimizing reordering features directly for BLEU with the decoder in the loop.
BibTeX
@inproceedings{cherry2013improved,
  author = {Colin Cherry},
  title = {{Improved Reordering for Phrase-Based Translation using Sparse Features}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2013},
  url = {https://openreview.net/forum?id=rybEwX-uZr},
}
Figure