What Can Syntax-Based MT Learn from Phrase-Based MT?

Steve DeNeefe, Kevin Knight, Wei Wang, Daniel MarcuEMNLP 2007

TL;DR

This paper compares syntax-based and phrase-based machine translation (MT) models, revealing that while syntax-based models produce more fluent output, they struggle with phrase coverage. The authors propose and evaluate methods to improve syntax-based models by incorporating phrase-based learning techniques, leading to significant gains in translation accuracy.

What Problem Does This Paper Address?

Statistical machine translation (SMT) has seen great success with two main families of approaches: phrase-based MT (PBMT) and syntax-based MT (SBMT). PBMT models, like the Alignment Template system (ATS), excel at learning a vast number of phrase pairs from parallel corpora, allowing for flexible reordering. However, their output can sometimes lack grammatical coherence. SBMT models, on the other hand, leverage linguistic structure (like parse trees) to generate more syntactically well-formed translations, but they often suffer from limited phrase coverage, meaning they miss many valid translation units. This paper aims to bridge this gap by investigating what SBMT can learn from the strengths of PBMT, specifically focusing on improving SBMT's phrase coverage and, consequently, its overall translation quality.

Methods

The paper compares two representative systems: the ATS for PBMT and the GHKM system for SBMT. The core of the paper lies in analyzing the phrase extraction mechanisms of both.

  • Phrase-based Extraction (ATS): This method extracts contiguous sequences of source and target words that are aligned in a parallel corpus. It enumerates all possible source phrases up to a certain length limit and identifies corresponding target phrases.
  • Syntax-based Extraction (GHKM): This method extracts translation rules that map source language sequences to target language syntax trees. These rules can be more complex, incorporating syntactic decorations, contextual constraints, non-constituent phrases, and even structural elements without words. A key characteristic is the extraction of "minimally-sized" rules that explain the training example.

The authors then explore several methods to improve GHKM's phrase coverage by drawing inspiration from PBMT:

  • Composed Rules: Instead of extracting only minimal rules, this approach allows combining multiple rules that have a parent-child relationship in the derivation tree to form larger, "composed" rules. This is controlled by a "size limit" parameter.
  • SPMT Model 1 Rules: This method, proposed by Marcu et al. (2006), is an alternative rule extraction technique that aims to capture larger translation units.
  • Restructuring Trees (Head-out Binarization): This technique modifies the target language syntax trees by binarizing them (breaking them down into smaller, binary structures). This aims to reduce syntactic constraints that might prevent the extraction of certain phrases.

Datasets

The study uses parallel corpora for Chinese-Arabic translation. Specifically, they mention:

  • Training Corpora: LDC2003E07/LDC2003E14 (Chinese), LDC2004T17/LDC2004T18 (Arabic), and LDC2005T06/LDC2005E46 (Chinese).
  • Evaluation Corpora: NIST 2003 MT evaluation (test set) and NIST 2002 MT evaluation (development set).

The datasets are described by their document IDs, number of segments, and number of words in each language.

Key Insights and Ideas

  • Complementary Strengths: PBMT excels at acquiring a broad range of phrase pairs, while SBMT produces more syntactically coherent output. Neither model is a strict superset of the other in terms of phrase coverage.
  • GHKM's Minimality Constraint is a Bottleneck: The GHKM system's tendency to extract only minimally-sized rules is a primary reason it misses many useful phrase pairs that ATS can learn.
  • Composed Rules Significantly Boost Coverage: Allowing GHKM to learn larger, composed rules (up to size 4) dramatically increases the number of ATS-useful phrase pairs it can extract.
  • Binarization Helps, But Has Trade-offs: Restructuring target trees through binarization also improves phrasal coverage but leads to a substantial increase in the total number of rules, potentially impacting decoding efficiency.
  • Syntax-Specific Rules Can Be Overly Constrained: GHKM can sometimes learn many specific rules for a single phrase translation (e.g., "prime minister"), whereas ATS might learn a single, more general rule. This can lead to situations where GHKM cannot translate certain phrases due to a lack of a perfectly matching contextual rule.

Key Results

  • Improved Phrasal Coverage: By applying composed rules (size 4) and SPMT model 1, the number of missing ATS-useful phrase pairs for GHKM was reduced from 1,994/2,199 (Chinese/Arabic) to 900/1,041.
  • Translation Accuracy Gains: The improvements in phrasal coverage directly translated to better translation quality. For Chinese, the BLEU score increased from a baseline GHKM of 38.02 to 42.04 with composed rules and SPMT model 1. For Arabic, it rose from 49.34 to 51.73.
  • Binarization Further Enhances Accuracy: Applying head-out binarization on top of composed rules and SPMT model 1 yielded the highest BLEU scores: 42.17 for Chinese and 52.50 for Arabic on the test set.
  • Overlap is Limited: Only about half of the phrase pairs extracted by ATS were also learned by GHKM, highlighting the fundamental differences in their extraction strategies.

Limitations and Future Work

  • Rule Explosion: The methods to improve phrasal coverage, particularly composed rules and binarization, lead to a significant increase in the number of rules. This can increase processing work and decoding times, suggesting diminishing returns at larger size limits.
  • Decoding Efficiency: While phrasal coverage and accuracy improve, the paper notes that decoding efficiency remains an area needing progress for syntax models.
  • Future Work: The authors suggest that incorporating syntactic knowledge and constraints into phrase-based models could also lead to improvements. For syntax models, further work on decoding efficiency and addressing remaining holes in phrasal coverage is needed. The paper also hints at exploring other solutions for handling unaligned words and improving the specificity of syntax rules.

Summary updated

BibTeX
@inproceedings{deneefe2007what,
  author = {Steve DeNeefe and Kevin Knight and Wei Wang and Daniel Marcu},
  title = {{What Can Syntax-Based MT Learn from Phrase-Based MT?}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2007},
}
Figure