TL;DR
This paper tackles the issue of syntax trees, commonly used in syntax-based machine translation, being too generic for effective translation. The authors propose relabeling these trees with more specific information, leading to a statistically significant improvement in translation quality as measured by BLEU scores.
What Problem Does This Paper Address?
Statistical Machine Translation (SMT) has evolved from phrase-based models to syntax-based approaches that leverage linguistic structure. Syntax-based SMT aims to enforce grammatical constraints and capture long-distance dependencies, offering potential advantages over simpler models. However, a key resource for these systems is the Penn Treebank (PTB), a corpus of English sentences annotated with syntactic parse trees. The authors argue that the PTB's tagset, while useful for general parsing, is too general for machine translation. This over-generality leads to grammatical errors in translations, such as incorrect determiner-noun agreement or verb tense mismatches, because the tags don't capture crucial distinctions needed for accurate translation.
Methods
The core contribution of this paper is the development and evaluation of various relabeling strategies for PTB syntax trees. These strategies aim to enrich the information contained within the tree labels, making them more suitable for MT. The authors categorize these strategies into two main types:
- Internal Annotation: This involves adding information about a node and its descendants to its surrounding nodes.
- Lexicalization: This technique annotates a lexical item (a word) onto an ancestor label, not just its parent. This helps capture word-specific behavior. They explore lexicalizing prepositions (LEX_PREP) and determiners (LEX_DT), among others.
- Tag Annotation: This partitions a grammatical category into more specific subcategories. For example, a Verb Phrase (VP) headed by a gerund (VBG) can be tagged as VP_VBG. This helps resolve issues like auxiliary/verb tense disagreement. They specifically use TAG_VP to distinguish verb forms.
- External Annotation: This involves passing information from the external environment into a node.
- Sisterhood Annotation: This annotates a nonterminal with information about its siblings (left, right, or both). This helps distinguish words based on their position within a constituent.
- Parent Annotation: This annotates a node with its parent's label. This provides context about the node's grammatical role within its parent structure.
- Complement Annotation: This annotates a node based on the type of complement it takes. This is useful for prepositions that have strong preferences for certain complement types.
They also explore Removal of Parser Annotations, investigating whether common parser additions (like NPB, -C suffixes, and SG tags) are beneficial for MT.
Datasets
- Training Data: 164M+167M words of parallel Chinese/English text.
- English Parsing: Collins' Model 2 (Collins, 1999) was used to parse the English side of the training data.
- Word Alignment: GIZA++ (Och and Ney, 2000) was used for word alignment.
- Test Set: 878-sentence 2002 NIST MT evaluation test set.
- Language Model Training: 800M words of English news text.
No new benchmarks were introduced; the evaluation was performed on an existing test set.
Key Insights and Ideas
- PTB is not MT-ready out-of-the-box: The generic nature of PTB tags is a significant impediment to high-quality machine translation, necessitating modifications.
- Specificity is key: Relabeling trees to be more specific about grammatical distinctions (e.g., determiner type, verb form, node position) directly leads to improved translation accuracy.
- Internal annotations are generally beneficial: Lexicalization and tag annotation strategies, which enrich node information, showed strong positive impacts on BLEU scores.
- External context matters: Sisterhood, parent, and complement annotations, which leverage contextual information, also proved effective, with sisterhood annotation being particularly impactful.
- Combinatorial effects: The most significant gains were achieved by combining multiple relabeling strategies, demonstrating that these improvements are additive.
- Parser annotations are helpful: Removing common parser annotations (like NPB, -C, SG) negatively impacted BLEU scores, suggesting these additions are valuable for MT.
Key Results
The paper reports a statistically significant improvement of 2.3 BLEU points over a baseline syntax-based system. This improvement was achieved by combining five of the most promising relabeling strategies (LEX_%, LEX_DT variant 2, TAG_VP, LEX_PREP variant 2, and SISTERHOOD variant 1) into an additive system, reaching a final BLEU score of 22.38. Figure 3 and Figure 11 provide detailed breakdowns of the impact of individual relabeling methods on ruleset size and BLEU score, showing significant increases for many strategies.
Limitations and Future Work
- Limitations:
- The paper acknowledges that some relabeling strategies, while helpful for parsing, proved detrimental to their MT system (e.g., LEX_CC).
- The additive system, while effective, still has room for improvement.
- The study focuses on English as the target language; the effectiveness of these strategies for other language pairs might vary.
- Future Work:
- The authors suggest exploring more sophisticated relabeling strategies.
- Investigating the interaction between different relabeling techniques and their optimal combinations remains an open area.
- Applying these relabeling techniques to other syntax-based MT frameworks or even phrase-based systems could be explored.
Summary updated