An Empirical Study of Translation Rule Extraction with Multiple Parsers

Tong Xiao, Jingbo Zhu, Hao Zhang, Muhua ZhuCOLING 2010

TL;DR

This paper tackles the challenge of low rule coverage in syntax-based machine translation (SMT) by proposing a novel method that leverages multiple parsers to extract translation rules. By combining rules from diverse parsing outputs, they significantly improve rule coverage and achieve a notable boost in translation accuracy, demonstrating that SMT systems are surprisingly robust to individual parser performance.

What Problem Does This Paper Address?

Syntax-based Statistical Machine Translation (SMT) systems rely on extracting translation rules from parallel corpora. A critical factor for their success is "rule coverage," which refers to how well the extracted rules can cover the linguistic phenomena present in the data. Traditional methods often extract rules from the single "best" parse tree generated by one parser. This approach, however, can lead to limited rule coverage because a single parser's output might not capture the full linguistic diversity. This limitation hinders the overall translation accuracy of syntax-based SMT systems. The paper aims to address this by exploring how to increase rule coverage through the use of multiple parsers.

Methods

The core of the paper's contribution lies in its method for improving rule coverage by using multiple parsers. Instead of relying on the output of a single parser, the authors propose to extract translation rules from the parse trees generated by several different parsers. The idea is that different parsers, due to their varying architectures and training data, will produce diverse parse trees, thereby leading to a richer and more comprehensive set of translation rules when combined.

The paper utilizes the standard GHKM extraction method for minimal rules and supplements it with SPMT models for phrasal rules. Rule composition techniques are also employed to create larger, more complex rules from smaller ones. The key novelty is the aggregation of rules extracted from the outputs of multiple parsers. They also introduce parser indicator features, which are trained using Minimum Error Rate Training (MERT) to allow the SMT system to learn preferences for rules derived from different parsers.

Datasets

The experiments were conducted on a Chinese-English parallel corpus consisting of 370K sentence pairs. This dataset was previously used in Section 4 of the paper. Word alignments were generated using GIZA++ and the inter-sect-diag-grow method. A 5-gram language model was trained on the target-side of the bilingual data and the Xinhua portion of the English Gigaword corpus. The development dataset came from the NIST MT 2003 evaluation set, and the test sets were from NIST MT evaluations in 2004 and 2005.

Key Insights and Ideas

  • Complementarity of Parsers: The most significant insight is the "complementarity" between rule sets extracted from different parsers. Even though individual parsers might have high performance, their outputs are diverse enough that combining their rule sets leads to substantial gains in coverage that wouldn't be achieved by simply improving a single parser.
  • Robustness to Parser Performance: Surprisingly, the paper finds that the SMT system's performance is not highly sensitive to the parsing performance of individual parsers. Even when parsers have significantly different F1 scores, the resulting MT systems show minimal differences in performance. This suggests that the diversity of rules extracted is more important than the absolute quality of any single parser's output.
  • Multi-parser as a General Improvement: The proposed multi-parser approach is shown to be compatible with existing techniques like rule composition and k-best parsing, suggesting it can be a general enhancement for syntax-based SMT systems.

Key Results

The paper demonstrates significant improvements in both rule coverage and translation accuracy:

  • Rule Coverage: Combining rules from multiple parsers led to substantial increases in rule coverage. For instance, combining two parsers increased coverage by over 20%, and using all four parsers together yielded gains of over 50% compared to single-parser baselines.
  • Translation Accuracy (BLEU Score): The multi-parser approach consistently improved BLEU scores. Using two parsers yielded a gain of over 0.4 BLEU points on the MT04 and MT05 test sets. This improvement increased to +0.7 BLEU with three parsers and reached a significant +0.9 BLEU improvement when all four parsers were used.
  • Comparison with k-best: The multi-parser approach outperformed k-best extraction (using up to 50-best parses from a single parser) by 0.6 BLEU points when using four parsers. However, the authors note that k-best extraction significantly increases extraction time, making it less practical.

Limitations and Future Work

  • Parser Training Data: All parsers used in the experiments were trained on the same treebank. The authors suggest that using parsers trained on different treebanks (e.g., Penn Treebank vs. ICE-GB) could provide even greater diversity. However, they acknowledge the challenge of incompatibility between different annotation standards and propose solutions like transforming rules into a unified form or using heterogeneous decoding techniques.
  • K-best Extraction Speed: While k-best extraction can improve results, its computational cost is a significant drawback, making it less appealing for practical applications.
  • Forest-based Extraction: The paper suggests that their multi-parser approach could be combined with forest-based rule extraction (which has shown to be effective) for potentially even greater improvements.

Summary updated

BibTeX
@inproceedings{xiao2010empirical,
  author = {Tong Xiao and Jingbo Zhu and Hao Zhang and Muhua Zhu},
  title = {{An Empirical Study of Translation Rule Extraction with Multiple Parsers}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2010},
}
Figure