Exploring Difficulties in Parsing Imperatives and Questions

Tadayoshi Hara, Takuya Matsuzaki, Yusuke Miyao, Jun'ichi TsujiiIJCNLP 2011 · OpenReview

Abstract: Tadayoshi Hara, Takuya Matsuzaki, Yusuke Miyao, Jun’ichi Tsujii. Proceedings of 5th International Joint Conference on Natural Language Processing.

2011.

Introduction

Natural language parsers, the tools used to determine the grammatical structure of sentences, have reached high levels of accuracy on standard news text, such as the Wall Street Journal (WSJ) portion of the Penn Treebank. However, their performance often degrades significantly when applied to "out-of-domain" text. While this drop is frequently attributed to lexical differences (new vocabulary), this research highlights the critical role of structural variation.

Specifically, the authors investigate imperatives (commands) and questions. These sentence types deviate from standard English syntax through omitted subjects ("Go home!") or inverted word orders ("Are you going?"). This paper explores whether standard supervised domain adaptation can bridge the performance gap for these unique syntactic structures.

Methodology and Datasets

The study evaluated three types of parsing technologies: shallow dependency parsers (MST and Malt), a deep HPSG-based parser (Enju), and a Part-of-Speech (POS) tagger. To address performance issues, the researchers employed supervised domain adaptation, using small amounts of annotated data from the target domains to refine models originally trained on WSJ news data.

The datasets used included:

  • The Brown Corpus (Fiction): 750 imperatives and 1,241 questions manually verified.
  • QuestionBank: 3,859 questions, primarily query-style.
  • WSJ (Penn Treebank): The baseline training data.
Examples of imperative and interrogative sentences from the Brown corpus, illustrating the syntactic challenges they present.

Key Insights and Results

The research identified several fundamental challenges in parsing these "non-standard" sentences:

  1. The POS Tagging Bottleneck: Standard taggers are biased toward news text where sentences typically start with nouns or determiners. In imperatives, the initial base-form verb is often misidentified as a noun, leading to cascading errors in the parse.
  2. Structural Misidentification: Parsers frequently failed to identify the main action (ROOT) of a sentence, sometimes mistaking vocatives (e.g., names) for the primary verb.
  3. The Data Ceiling: Learning curves indicated that performance gains from additional training data began to plateau after 300–400 sentences. This suggests that the issue lies within the model architecture itself, which relies heavily on local word patterns (N-grams) rather than global structural understanding.
POS tagging accuracy across different sentence types. Even with adaptation, accuracy for imperatives and questions remains below the standard news baseline.

While adaptation improved accuracy—notably by 25–35% for QuestionBank—final scores remained significantly lower than the 97%+ accuracy typical for news text. Furthermore, a model adapted to one type of question (e.g., wh- questions in QuestionBank) performed poorly on others (e.g., yes/no questions in Brown), indicating that these categories are not syntactically monolithic.

Conclusion

The study demonstrates that current parsing models are fundamentally ill-equipped for the global structural shifts found in imperatives and questions. Future work should focus on incorporating richer features, such as dialogue context, and developing frameworks more robust to constituent reordering. Additionally, the findings suggest that "gold standard" corpora may require cleaning to better reflect the nuances of spoken-style and fictional text.

Summary updated

BibTeX
@inproceedings{hara2011exploring,
  author = {Tadayoshi Hara and Takuya Matsuzaki and Yusuke Miyao and Jun'ichi Tsujii},
  title = {{Exploring Difficulties in Parsing Imperatives and Questions}},
  booktitle = {IJCNLP},
  year = {2011},
  url = {https://openreview.net/forum?id=vsZkMGHul-X},
}
Figure