Syntactic discriminative language model rerankers for statistical machine translation

Simon Carter, Christof MonzMach. Transl 2011 · OpenReview

Abstract This article describes a method that successfully exploits syntactic features for n-best translation candidate reranking using perceptrons. We motivate the utility of syntax by demonstrating the superior performance of parsers over n-gram language models in differentiating between Statistical Machine Translation output and human translations. Our approach uses discriminative language modelling to rerank the n-best translations generated by a statistical machine translation system. The performance is evaluated for Arabic-to-English translation using NIST’s MT-Eval benchmarks. While deep features extracted from parse trees do not consistently help, we show how features extracted from a shallow Part-of-Speech annotation layer outperform a competitive baseline and a state-of-the-art comparative reranking approach, leading to significant BLEU improvements on three different test sets.

Introduction

Statistical Machine Translation (SMT) systems traditionally rely on n-gram language models to ensure the output sounds fluent. However, n-grams have a "short memory," usually only looking at the last 3 to 5 words. This means they often miss long-range structural errors—like a verb appearing at the wrong end of a long sentence. While researchers have long suspected that syntax (the formal rules of sentence structure) could solve this, simply plugging a standard parser into a translation system rarely works.

Methodology

The authors propose a Syntactic Discriminative Language Model (DLM) for reranking the "n-best" list of translation candidates produced by an SMT system. The framework utilizes a Perceptron algorithm, an online learner that adjusts feature weights based on whether a candidate translation's score matches its actual quality (measured by BLEU score).

Feature Extraction

  1. Deep Features: Complex features from full parse trees (using Collins Model 2), including head-parent relationships, tree depth, and "unit productions."
  2. Shallow Features: Part-of-Speech (POS) n-grams.
  3. Simple POS (S-POS) Tagger: Crucially, the authors introduced a context-blind tagger that assigns each word its most frequent tag from a training set. The goal was to see if a "dumber" tagger would be less likely to "smooth over" the reordering mistakes made by the translation engine.

Key Insights and Ideas

Parsers as "Human-Detectors"

Initial experiments showed that parsers are significantly more accurate than 4-gram models at identifying which sentence was written by a human versus a machine. This motivates the use of syntactic information, as parsers capture structural properties that n-grams ignore.

Figure (b): Model accuracy in differentiating human vs. machine translations across BLEU ranges (MT04).

The "Forgiveness" Trap

Despite being good at identifying humans, parsers struggle to rank different machine outputs. Because they are trained to find structure, they often "hallucinate" a valid grammatical path through a messy sentence (e.g., tagging a noun as a verb just to make a tree fit), which masks the translation's poor quality. This advantage of parsers over n-grams is consistent across various sentence lengths, as shown below.

Figure (f): Model accuracy in differentiating human vs. machine translations across sentence lengths (MT04).

Simplicity Wins

The most surprising finding was that the S-POS tagger outperformed sophisticated CRF taggers and deep parse features. Because the S-POS tagger is context-blind, it doesn't try to "fix" a broken sentence. If a word is in a weird spot, the resulting POS sequence looks "wrong" to the discriminative model, making it easier to penalize.

Key Results

  • BLEU Score Improvements: The S-POS reranker achieved statistically significant improvements across all three test sets compared to a competitive n-gram baseline. On MT06, the S-POS model improved the BLEU score by 0.40 points over the n-gram reranker.
  • Precision Gains: Qualitative analysis showed that the syntactic features specifically helped with word reordering. The models showed higher precision for bigrams, trigrams, and four-grams, indicating that the "flow" of the sentences was structurally sounder.
  • Robustness: While deep features were hit-or-miss, the shallow S-POS features provided consistent gains, proving more robust for real-world SMT applications.

Conclusion

Full parsers are computationally expensive and often fail to produce any parse for very long or highly disfluent sentences. Furthermore, the massive feature space created by deep syntax makes it prone to overfitting. The authors suggest that shallow syntactic features, like those from the S-POS tagger, provide a more robust and effective middle ground for improving machine translation quality.

Summary updated

Abstract

This article describes a method that successfully exploits syntactic features for n-best translation candidate reranking using perceptrons. We motivate the utility of syntax by demonstrating the superior performance of parsers over n-gram language models in differentiating between Statistical Machine Translation output and human translations. Our approach uses discriminative language modelling to rerank the n-best translations generated by a statistical machine translation system. The performance is evaluated for Arabic-to-English translation using NIST’s MT-Eval benchmarks. While deep features extracted from parse trees do not consistently help, we show how features extracted from a shallow Part-of-Speech annotation layer outperform a competitive baseline and a state-of-the-art comparative reranking approach, leading to significant BLEU improvements on three different test sets.
BibTeX
@inproceedings{carter2011syntactic,
  author = {Simon Carter and Christof Monz},
  title = {{Syntactic discriminative language model rerankers for statistical machine translation}},
  booktitle = {Mach. Transl},
  year = {2011},
  url = {https://openreview.net/forum?id=Y9SFKUbeLSL},
}
Figure