Grammatical Machine Translation

Stefan Riezler, John T. Maxwell IiiNAACL 2006

TL;DR

This paper introduces a hybrid machine translation system that combines the strengths of statistical phrase-based methods with traditional grammar-based approaches. By integrating a grammar-based generator into a dependency-based SMT framework, the system achieves state-of-the-art translation quality on in-coverage examples while significantly improving grammatical accuracy.

What Problem Does This Paper Address?

Statistical Machine Translation (SMT), particularly phrase-based SMT, excels at modeling local word order and idiomatic expressions. However, it struggles with long-distance dependencies and generalizing to unseen phrases that share underlying linguistic structures. Traditional grammar-based MT, while good at capturing grammatical structure, can be brittle and difficult to scale. This paper aims to bridge this gap by leveraging syntactic knowledge to improve SMT's grammatical accuracy and generalization capabilities, without sacrificing overall translation quality.

Methods

The proposed system integrates ideas from both phrase-based SMT and grammar-based MT. It uses a transfer system that operates on "dependency structure snippets" derived from functional (f-)structures. These snippets are extracted using a process that involves:

  • Improved Word Alignment: Intersecting alignment matrices from both translation directions and refining them.
  • f-structure Parsing: Parsing source and target sentences using LFG grammars to obtain f-structures.
  • Snippet Extraction: Identifying and aligning substructures (snippets) within the f-structures that correspond to multi-word translation units.
  • Primitive Transfer Rule Induction: Extracting basic rules for mapping lexical predicates and local f-structure configurations.
  • Complex Transfer Rule Combination: Combining primitive rules to handle more complex mappings, with a limit on the number of combined rules to manage complexity.

Crucially, the system then feeds these transferred f-structure snippets into a grammar-based generator to produce the target sentence. Unlike previous approaches that focused on statistical ordering models after translation, this method prioritizes grammatical generation. Statistical components (like rule probabilities and language models) are trained using minimum error rate training, similar to state-of-the-art SMT systems.

Datasets

  • Training Data: The Europarl parallel corpus was used. For quick experimentation, sentences with 5 to 15 words were selected, resulting in a training set of 163,141 sentences.
  • Development Set: 1967 sentences were used for development.
  • Test Set: 1,755 sentences of length 5-15 from the Europarl corpus, previously used in Koehn et al. (2003), were used for final evaluation.
  • Grammars: LFG grammars for German and English were used for parsing.

Key Insights and Ideas

  • Grammar as a Generator, Not Just a Parser: The core innovation is using a grammar-based generator after the statistical transfer process, rather than solely relying on statistical models for ordering. This directly addresses the grammaticality issue.
  • Dependency Structure Snippets: Breaking down f-structures into manageable "snippets" allows for a more flexible transfer mechanism that can capture more complex linguistic relationships than simple phrase pairs.
  • Hybrid System Potential: The ability to determine when the system is "in-coverage" (i.e., when the grammar can handle the generated structure) opens the door for a hybrid system that combines the best of both worlds: state-of-the-art quality for in-coverage cases and a fallback for out-of-coverage ones.
  • Grammaticality vs. Adequacy Trade-off: The paper highlights that while NIST scores might not show significant gains on the full test set (due to out-of-coverage issues), manual evaluations reveal substantial improvements in grammaticality and adequacy for in-coverage examples.

Key Results

  • In-Coverage NIST Scores: Achieved results comparable to state-of-the-art phrase-based SMT systems (Koehn et al., 2003) on in-coverage examples.
  • Manual Evaluation (In-Coverage):
    • Judges showed a preference for the LFG-based system over the phrase-based system for both adequacy and grammaticality.
  • Net improvement in translational adequacy of 11.4% (57 sentences) and grammaticality of 15.4% (77 sentences) over 500 in-coverage examples.
  • Manual Evaluation (Out-of-Coverage): For out-of-coverage examples, the phrase-based system was generally preferred.
  • Overall: The system provides improved grammaticality on in-coverage examples, suggesting a path towards hybrid systems that maintain high translation quality.

Limitations and Future Work

  • Out-of-Coverage Performance: The system's performance on out-of-coverage examples is suboptimal, leading to a statistically insignificant difference in NIST scores on the full test set compared to IBM model 4. This is attributed to errors in parsing (20% FRAGMENT parses) and rule extraction, forcing the generator to rely on robustness techniques.
  • Integration Complexity: The integration of transfer and generation is described as suboptimal, making it difficult to work with large n-best lists during training and application.
  • Future Work:
    • Improve in-coverage translations, potentially through stochastic generation.
  • Apply the system to other language pairs and larger datasets.
  • Explore improvements to the parse-transfer-generation pipeline to mitigate error propagation.

Summary updated

BibTeX
@inproceedings{riezler2006grammatical,
  author = {Stefan Riezler and John T. Maxwell III},
  title = {{Grammatical Machine Translation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2006},
}
Figure