Improving Syntax-Augmented Machine Translation by Coarsening the Label Set

Greg Hanneman, Alon LavieNAACL 2013

TL;DR

This paper introduces a method to simplify complex syntax-augmented machine translation (SAMT) grammars by "coarsening" their label sets. By reducing the number of nonterminal labels, they achieve improved translation quality and mitigate common issues like sparsity and ambiguity, making SAMT systems more efficient and robust.

What Problem Does This Paper Address?

Statistical Machine Translation (SMT) systems that leverage linguistic syntax (like Syntax-Augmented Machine Translation, or SAMT) often use very large sets of nonterminal labels. While these rich labels allow for more precise linguistic representation and can improve translation coverage, they also lead to significant problems:

  • Increased Grammar Size: More labels mean larger grammars, which are computationally expensive to process.
  • Derivational Ambiguity: The same translation can be derived in many different ways due to the numerous label combinations, leading to redundant hypotheses and slower decoding.
  • Data Sparsity: With a vast number of labels, it becomes less likely that specific label combinations will be seen in the training data, leading to "rule sparsity" where the system lacks rules for certain linguistic constructions.

The authors aim to address these issues by reducing the complexity of SAMT grammars without sacrificing their expressive power.

Methods

The core contribution is an adaptation of a "label collapsing" algorithm, originally developed for tree-to-tree grammars, to the SAMT formalism. Here's how it works:

  1. Bilingual Labeling: They first extend standard SAMT grammars (which are typically monolingual on the target side) to have bilingual labels. This means each nonterminal label represents a pair of source and target language syntactic categories (e.g., VBD::VV). This is achieved by modifying an existing grammar extraction framework (Thrax) to handle source-side parsing.
  2. Label Collapsing Algorithm: The bilingual labels are then fed into the label collapsing algorithm. This algorithm works by:
  3. Calculating the probability of a source label appearing with a target label, and vice-versa.
  4. Using these probabilities to define a distance metric between monolingual labels.
  5. Employing an agglomerative clustering approach to iteratively merge the closest pairs of labels. This process groups similar labels together, effectively coarsening the label set.
  6. Source Label Removal: After the collapsing process, the source-side labels are discarded, resulting in a new SAMT grammar with a smaller set of monolingual (target-side) labels. This new grammar retains the same expressive power but is significantly more compact.

Datasets

  • Training Data: The experiments use the FBIS corpus, consisting of approximately 302,000 parallel Chinese-English sentence pairs.
  • Tuning Data: The NIST Open MT 2006 data set (1664 sentences) was used for tuning the MT systems.
  • Test Data:
    • NIST 2003 data set (919 sentences).
  • NIST 2008 data set (1357 sentences).
  • All tuning and test sets had four English references.

Key Insights and Ideas

  • Coarsening as a Solution: The paper demonstrates that simplifying complex grammars through label coarsening is a viable and effective strategy for improving MT performance and addressing common modeling challenges.
  • Bilingual Labels are Key: The ability to induce bilingual labels is crucial for applying the label collapsing algorithm effectively to SAMT grammars, as it allows for the calculation of the necessary conditional probabilities.
  • Impact on Sparsity and Ambiguity: The reduction in label set size directly combats rule sparsity and derivational ambiguity, leading to more robust and efficient translation systems.
  • Robustness to Domain Shift: The coarsened grammars appear to be more robust to test-set or domain variations compared to uncollapsed SAMT or single-nonterminal Hiero grammars.
  • Hierarchical Rules Benefit Most: Label collapsing disproportionately affects hierarchical rules, which are more prone to label variations, leading to a more streamlined grammar.

Key Results

  • Translation Quality Improvement: The "Collapse 3" system (with 72 target-side labels) achieved significant improvements:
  • 1.14 BLEU points higher on MT '03 compared to the baseline SAMT.
  • 1.51 BLEU points higher on MT '08 compared to the baseline SAMT.
  • Comparison to Hiero: The collapsed systems also outperformed a Hiero baseline (which uses a single nonterminal) on the MT '08 test set, suggesting that even with coarsening, the remaining syntactic structure is beneficial.
  • Grammar Size Reduction: The number of target-side labels was drastically reduced, from 4181 in the baseline SAMT to as few as 72 in the "Collapse 3" system.
  • Reduced Ambiguity and Sparsity: Analysis showed significant reductions in derivational ambiguity and rule sparsity, as evidenced by the more compact grammars and increased application of hierarchical rules.

Limitations and Future Work

  • Collapsing Strategy: The current implementation indiscriminately allows either source or target labels to be collapsed. The authors suggest that for SAMT, where source labels are dropped in the final grammar, it might be more intuitive to only collapse target-side labels.
  • Stopping Criterion: The paper mentions that manually identifying the optimal stopping point for the collapsing algorithm is becoming infeasible with the large number of iterations. Future work will focus on developing an automatic stopping criterion or a more principled test for determining when further collapsing yields diminishing returns.

Summary updated

BibTeX
@inproceedings{hanneman2013improving,
  author = {Greg Hanneman and Alon Lavie},
  title = {{Improving Syntax-Augmented Machine Translation by Coarsening the Label Set}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2013},
}
Figure