Discriminative Alignment Training without Annotated Data for Machine Translation

Patrik Lambert, Rafael E. Banchs, Josep M. CregoNAACL 2007

TL;DR

This paper introduces a novel framework for training word alignment models for machine translation (MT) that directly optimizes for translation quality, rather than relying on separate, pre-trained alignment models. This approach bypasses the need for word-level annotated data, using only sentence-level references and automated translation metrics for optimization.

What Problem Does This Paper Address?

Statistical Machine Translation (SMT) systems typically train alignment models as a separate, preliminary step before training the translation model itself. This means the alignment parameters are not directly optimized to improve the final translation output. While methods exist to tune the translation model's feature weights using development sets with reference translations, the alignment stage remains somewhat detached. Existing alignment training methods, like unsupervised EM-based approaches, can be computationally expensive and difficult to extend with new features. Supervised methods, while easier to extend, require costly word-level annotations. Furthermore, there's a disconnect between traditional alignment evaluation metrics (like AER) and actual translation quality, as different MT system types benefit from different alignment characteristics (e.g., precision vs. recall). This paper aims to bridge this gap by developing a discriminative alignment training framework that directly optimizes for translation quality using automated metrics, without requiring word-level annotations.

Methods

The core of this paper is a novel framework for discriminative alignment training. Instead of training alignment models independently, they are trained to optimize a downstream MT system's performance. The key innovation is using automated translation evaluation metrics (like BLEU, NIST) as the objective function for alignment optimization. This means the alignment model's parameters are adjusted to produce alignments that lead to better translations, as judged by these metrics.

The authors implement a Bilingual Word Aligner (BIA), which is a beam-search decoder. This aligner is designed with specific features to address challenges in extracting bilingual units for N-gram SMT:

  • Unlinked word penalties: Differentiates penalties for unlinked words in the source vs. target language.
  • Link bonus: Encourages more links in the alignment to favor higher recall, beneficial for N-gram SMT.
  • Embedded word position penalty: Penalizes alignments where word positions are "embedded" between links, which can lead to data sparseness in the translation model.
  • Distortion features: Similar to existing methods, these penalize reorderings in alignments.
  • $2$ score: Used as a word and POS-tag association model.

The optimization procedure uses the Stochastic Perturbation Stochastic Approximation (SPSA) algorithm to iteratively tune the alignment model's coefficients. This algorithm is chosen for its robustness and efficiency, requiring only two objective function evaluations per iteration.

Datasets

  • Training Corpus: Chinese-English data from the IWSLT'06 evaluation campaign, containing 46,000 sentences.
  • Development Set (dev4): Used for tuning alignment coefficients and SMT model weights. It consists of 489 sentences with 7 references.
  • Test Set: A selection of 500 sentences from dev1, dev2, and dev3 sets, with an average of 6 words per sentence and 16 references.

Key Insights and Ideas

  • Directly optimizing alignment for translation quality is feasible and beneficial: By using automated translation metrics as the objective, the alignment process becomes directly aligned with the ultimate goal of producing better translations.
  • Eliminating the need for word-level annotations: This is a significant practical advantage, as obtaining such annotations is expensive and time-consuming. Sentence-level references are sufficient.
  • Alignment characteristics matter for different MT systems: The paper highlights that different MT architectures (e.g., phrase-based vs. N-gram SMT) benefit from different alignment properties (precision vs. recall), and their framework can tune for these specific needs.
  • The proposed method can outperform standard alignment techniques: The results suggest that alignments trained discriminatively with this approach lead to better translation quality compared to those produced by standard GIZA++ alignments.

Key Results

The paper reports that translation systems trained using BIA alignments optimized with their framework achieved better average scores across multiple evaluation metrics (BLEU, NIST, PER, WER) compared to systems trained with GIZA++ alignments.

Specifically, the BIA (BLEU) and BIA (BLEU+4*NIST) tuned alignments yielded the best results in terms of BLEU and WER, respectively. For instance, BIA (BLEU+4*NIST) achieved a BLEU score of 47.0 (0.4) and a WER of 40.9 (0.5), outperforming the best GIZA++ result of 43.7 (0.9) for BLEU and 43.7 (0.4) for WER.

Interestingly, tuning BIA with NIST alone resulted in a worse NIST score for the translation system, highlighting the complex interplay between alignment metrics and translation quality.

Limitations and Future Work

  • Computational Cost: The current approach requires aligning the entire training corpus at each iteration, making it computationally expensive for very large corpora.
  • Basic SMT Features: The initial version of the alignment system uses basic SMT models. Future work could incorporate more sophisticated SMT features, especially those relevant for phrase-based SMT systems.
  • Association Score Model: The authors suggest improving the association score model by adding discount factors or more association score types/dictionaries.
  • Exploring other SMT features: Investigating the use of various SMT features during the alignment coefficient optimization is a promising direction.
  • Scalability: Determining methods to make the approach scalable to large corpora is a key area for future research.

Summary updated

BibTeX
@inproceedings{lambert2007discriminative,
  author = {Patrik Lambert and Rafael E. Banchs and Josep M. Crego},
  title = {{Discriminative Alignment Training without Annotated Data for Machine Translation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2007},
}
Figure