A Discriminative Latent Variable Model for Statistical Machine Translation

Phil Blunsom, Trevor Cohn, Miles OsborneACL 2008 · OpenReview

Abstract

Large-scale discriminative machine translation promises to further the state-of-the-art, but has failed to deliver convincing gains over current heuristic frequency count systems. We argue that a principle reason for this failure is not dealing with multiple, equivalent translations. We present a translation model which models derivations as a latent variable, in both training and decoding, and is fully discriminative and globally optimised. Results show that accounting for multiple derivations does indeed improve performance. Additionally, we show that regularisation is essential for maximum conditional likelihood models in order to avoid degenerate solutions.

Introduction

In Statistical Machine Translation (SMT), the primary goal is to identify the optimal translation $e$ for a given source sentence $f$. Historically, most systems relied on generative models built from heuristic frequency counts. While discriminative models offer the advantage of incorporating thousands of overlapping, non-independent features, they have historically struggled to outperform simpler generative baselines.

The authors identify two primary obstacles to the success of discriminative SMT:

  1. Spurious Ambiguity: There are often thousands of distinct ways—derivations—to produce the same translation using grammar rules or phrase pairs. Most models arbitrarily select a single "reference" derivation for training, ignoring the vast amount of information contained in alternative derivations.
  2. Degenerate Solutions: Without proper constraints, powerful discriminative models tend to overfit by finding complex, obscure derivations that perfectly match training data but fail to generalize.
The number of derivations grows exponentially with sentence length, highlighting the challenge of spurious ambiguity.

Methodology

The proposed solution is a Discriminative Latent Variable Model based on a Synchronous Context-Free Grammar (SCFG).

Latent Derivations and Marginalization

Instead of maximizing the probability of a specific derivation, the model maximizes the conditional likelihood of the translation itself, $p(e|f)$. This is achieved by treating the derivation as a latent variable and "marginalizing it out"—summing the probabilities of every possible derivation that generates the target sentence. This approach treats the derivation as a nuisance variable, focusing on the end result rather than the specific path taken.

Global Training and Packed Charts

Training is performed on a log-linear model using a large parallel corpus. To manage the computational complexity of summing over an exponential number of derivations, the authors utilize the Inside-Outside algorithm over packed charts. These charts are efficient hypergraph representations that store all possible parses for a given sentence pair.

A packed chart (hypergraph) representing multiple possible derivations for the source 'sur la table'.

Regularization

To prevent the model from converging on degenerate solutions, a Gaussian prior (L2 regularization) is applied to the model parameters. This encourages the model to distribute probability mass across multiple derivations, effectively learning the inherent ambiguity of the translation task rather than overfitting to a single path.

Max-Translation Decoding

During inference, the model moves beyond the standard Viterbi algorithm (which finds the single best derivation). Instead, it employs a beam search to approximate the translation with the highest total probability across all its possible derivations, a process known as max-translation decoding.

Experimental Results

The model was evaluated using the Europarl V2 French-English corpus, with a training set of 170,000 sentence pairs (filtered for lengths 5–15) and a grammar of 7.8 million rules.

Key Findings

  • Marginalization Gains: On the development set, the "All Derivations" approach achieved 31.23 BLEU, significantly higher than the 27.32 BLEU obtained by training on a single derivation.
  • Decoding Performance: The discriminative max-translation model (27.72 BLEU) outperformed the max-derivation (Viterbi) model (25.78 BLEU) in head-to-head tests.
  • Competitive Baselines: Even with a minimalist feature set, the model performed comparably to the state-of-the-art "Hiero" system, demonstrating the power of the discriminative framework.

Discussion and Future Work

A key insight from this work is the "weight of evidence": multiple weak rules can collectively outvote a single high-scoring but incorrect rule. By summing over derivations, the model leverages the collective evidence of the entire grammar.

However, challenges remain. Approximately 24% of training sentences were discarded because the grammar could not "reach" the reference translation. Furthermore, the computational complexity of marginalization remains high. Future work will focus on scaling the model to longer sentences and incorporating more expressive features, such as source-side context and linguistic syntax, to fully exploit the potential of discriminative modeling.

Summary updated

Abstract

Large-scale discriminative machine translation promises to further the state-of-the-art, but has failed to deliver convincing gains over current heuristic frequency count systems. We argue that a principle reason for this failure is not dealing with multiple, equivalent translations. We present a translation model which models derivations as a latent variable, in both training and decoding, and is fully discriminative and globally optimised. Results show that accounting for multiple derivations does indeed improve performance. Additionally, we show that regularisation is essential for maximum conditional likelihood models in order to avoid degenerate solutions.
BibTeX
@inproceedings{blunsom2008discriminative,
  author = {Phil Blunsom and Trevor Cohn and Miles Osborne},
  title = {{A Discriminative Latent Variable Model for Statistical Machine Translation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2008},
  url = {https://openreview.net/forum?id=HkNlxTlOWS},
}
Figure