Abstract: Tadayoshi Hara, Takuya Matsuzaki, Yusuke Miyao, Jun’ichi Tsujii. Proceedings of 5th International Joint Conference on Natural Language Processing.
2011.
Introduction
Natural language parsers, the tools used to determine the grammatical structure of sentences, have reached high levels of accuracy on standard news text, such as the Wall Street Journal (WSJ) portion of the Penn Treebank. However, their performance often degrades significantly when applied to "out-of-domain" text. While this drop is frequently attributed to lexical differences (new vocabulary), this research highlights the critical role of structural variation.
Specifically, the authors investigate imperatives (commands) and questions. These sentence types deviate from standard English syntax through omitted subjects ("Go home!") or inverted word orders ("Are you going?"). This paper explores whether standard supervised domain adaptation can bridge the performance gap for these unique syntactic structures.
Methodology and Datasets
The study evaluated three types of parsing technologies: shallow dependency parsers (MST and Malt), a deep HPSG-based parser (Enju), and a Part-of-Speech (POS) tagger. To address performance issues, the researchers employed supervised domain adaptation, using small amounts of annotated data from the target domains to refine models originally trained on WSJ news data.
The datasets used included:
- The Brown Corpus (Fiction): 750 imperatives and 1,241 questions manually verified.
- QuestionBank: 3,859 questions, primarily query-style.
- WSJ (Penn Treebank): The baseline training data.
Key Insights and Results
The research identified several fundamental challenges in parsing these "non-standard" sentences:
- The POS Tagging Bottleneck: Standard taggers are biased toward news text where sentences typically start with nouns or determiners. In imperatives, the initial base-form verb is often misidentified as a noun, leading to cascading errors in the parse.
- Structural Misidentification: Parsers frequently failed to identify the main action (ROOT) of a sentence, sometimes mistaking vocatives (e.g., names) for the primary verb.
- The Data Ceiling: Learning curves indicated that performance gains from additional training data began to plateau after 300–400 sentences. This suggests that the issue lies within the model architecture itself, which relies heavily on local word patterns (N-grams) rather than global structural understanding.
While adaptation improved accuracy—notably by 25–35% for QuestionBank—final scores remained significantly lower than the 97%+ accuracy typical for news text. Furthermore, a model adapted to one type of question (e.g., wh- questions in QuestionBank) performed poorly on others (e.g., yes/no questions in Brown), indicating that these categories are not syntactically monolithic.
Conclusion
The study demonstrates that current parsing models are fundamentally ill-equipped for the global structural shifts found in imperatives and questions. Future work should focus on incorporating richer features, such as dialogue context, and developing frameworks more robust to constituent reordering. Additionally, the findings suggest that "gold standard" corpora may require cleaning to better reflect the nuances of spoken-style and fictional text.
Summary updated