Combining Morphosyntactic Enriched Representation with n-best Reranking in Statistical Translation

Hélène Bonneau-Maynard, Alexandre Allauzen, Daniel Déchelotte, Holger Schwenk

Introduction

In the mid-2000s, phrase-based Statistical Machine Translation (SMT) represented the state-of-the-art. While these models effectively captured local dependencies through statistical counts of word sequences, they often lacked explicit structural rules. This frequently resulted in grammatically poor outputs, characterized by incorrect word order or agreement errors, as standard $n$-gram language models struggle with long-term linguistic dependencies.

This paper proposes a novel approach to integrate morphosyntactic information—specifically Part-of-Speech (POS) tags—directly into the SMT pipeline. By creating "enriched units" where words are concatenated with their POS tags, the authors force the translation model to consider syntax during the generation phase, rather than relying solely on post-processing reranking.

Methodology

The core innovation lies in shifting morphosyntactic data from a secondary reranking step into the primary translation model:

  • Enriched Translation Units: The parallel corpus is pre-processed using TreeTagger. Each word is appended with its POS tag (e.g., European_NP Parliament_NP). The Moses SMT decoder is then trained on these enriched units.
  • Decoding and Post-processing: The system generates enriched sentences. The POS tags are stripped away only at the final stage to produce the standard word sequence for evaluation.
  • $n$-best Reranking: To further refine output, the decoder generates a 1000-best list of hypotheses. These are reranked using a log-linear combination of features, including higher-order (4-gram and 5-gram) word and POS language models.
  • Optimization: Feature weights for reranking are optimized to maximize BLEU scores using the Condor numerical optimization tool.

Experimental Setup

The researchers utilized the TC-STAR EPPS Corpus, consisting of transcriptions from European Parliament Plenary Sessions, for an English-to-Spanish translation task.

Table 1: Statistics of the TC-STAR EPPS parallel corpus used for training and evaluation.

As shown in the corpus statistics, the training set initially comprised 1.2 million sentence pairs. To ensure efficient alignment, sentences exceeding 40 words were filtered out, resulting in a training set of approximately 910,000 pairs. The monolingual Spanish corpus used for language modeling contained over 34 million words.

Key Results and Analysis

The experiments demonstrated that early integration of syntax significantly outperforms simple post-processing. Reranking a standard word-based model with a POS language model yielded negligible improvements, as the initial $n$-best list often lacked syntactically sound candidates.

In contrast, the enriched unit approach forced the model to generate better hypotheses from the start. For instance, the enriched model corrected gender agreement errors (e.g., changing "los últimos sesiones" to "los últimos períodos de sesiones") by leveraging the syntactic context provided by the tags.

Table 2: BLEU score results for various configurations of enriched units and reranking strategies.

The best performance was achieved by combining the enriched model with a 5-gram word LM and a 5-gram POS LM (using a 100-word stoplist), reaching a BLEU score of 46.8 on the test set—a 1.9% relative improvement over the baseline.

Conclusion and Future Work

Integrating morphosyntactic information directly into translation units proves more effective than using it solely for reranking. However, the authors noted limitations in the current POS tagger, which lacked gender and number distinctions—critical for highly inflected languages like Spanish. Future work aims to incorporate these features and move toward fully "factored translation models" to improve generalization across morphological variants.

Summary updated

Abstract

The purpose of this work is to explore the integration of morphosyntactic information into the translation model itself, by enriching words with their morphosyntactic categories. We investigate word disambiguation using morphosyntactic categories, n-best hypotheses reranking, and the combination of both methods with word or morphosyntactic n-gram language model reranking. Experiments are carried out on the English-to-Spanish translation task. Using the morphosyntactic language model alone does not results in any improvement in performance. However, combining morphosyntactic word disambiguation with a word based 4-gram language model results in a relative improvement in the BLEU score of 2.3% on the development set and 1.9% on the test set.
BibTeX
@inproceedings{bonneaumaynard2007combining,
  author = {Hélène Bonneau-Maynard and Alexandre Allauzen and Daniel Déchelotte and Holger Schwenk},
  title = {{Combining Morphosyntactic Enriched Representation with n-best Reranking in Statistical Translation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2007},
}
Figure