Effective Analysis of Causes and Inter-dependencies of Parsing Errors

Tadayoshi Hara, Yusuke Miyao, Jun'ichi TsujiiIWPT 2009 · OpenReview

Standard metrics like F-scores are the industry standard for evaluating Natural Language Processing (NLP) parsers, but they suffer from a significant limitation: they are purely quantitative. While they indicate how often a parser fails, they provide little insight into why it failed or how a developer might address the underlying linguistic issues. This paper, presented at IWPT 2009, introduces a diagnostic toolkit designed to open this "black box" by classifying errors and mapping their inter-dependencies.

The Challenge of Deep Parsing

Deep parsers aim to capture the underlying logical structure of a sentence—identifying predicate-argument relations (triplets) rather than just surface-level word order. Because these structures are hierarchical and interconnected, errors are rarely isolated. A single mistake in identifying a verb's subject can trigger a cascade of secondary errors. Manually untangling these dependencies is prohibitively time-consuming for developers.

Methodology: A Two-Pronged Approach

The authors propose two primary methods to automate error analysis and move toward systematic grammar improvement:

1. Pattern-based Classification

The system uses 14 predefined "error patterns" rooted in linguistic theory to categorize raw data errors into actionable labels. For example, if a parser fails to correctly attach a prepositional phrase, it is automatically flagged as a "Prepositional Attachment" error.

A comparison between a correct parse and a parser error involving prepositional attachment.

This categorization allows developers to see which specific linguistic phenomena—such as coordination, apposition, or relative clauses—are most problematic for the model.

2. Inter-dependency Analysis (Re-parsing)

To determine if one error caused another, the researchers developed a "force-correction" technique. By manually fixing one error and re-parsing the sentence, they could observe if other errors disappeared. If they did, those errors were labeled as dependent.

Analyzing whether errors occur independently or are mutually exclusive (conflicting).

This method groups errors into "propagation networks," helping developers identify "root cause" errors that, if fixed, would yield the highest improvement in overall accuracy.

Key Insights and Results

The researchers tested their toolkit using the Enju parser on the Penn Treebank (financial news) and the GENIA corpus (biomedical text).

  • The Correction Effect: The impact of fixing an error varies wildly by category. Fixing a "To-infinitive" error resolved an average of five other related errors, whereas fixing a "Prepositional Attachment" error rarely resolved anything else.
  • Independence Rates: Some errors are "loners." Prepositional attachment mistakes were independent 74.8% of the time. Conversely, errors in "Head selection for noun phrases" were found to have 0% independence—they are always symptoms of broader structural failures.
  • Domain Adaptation: When moving from news to biomedical text, the error rate increased from 10.48% to 14.04%. The toolkit pinpointed "Prepositional Attachment" and "Coordination" as the primary drivers of this decrease, providing a clear roadmap for domain-specific tuning.

Conclusion

By classifying over 40% of all parsing errors and identifying nearly 2,000 inter-dependent error groups, this research provides a systematic way to improve grammar-based parsers. While some errors remain "uncorrectable" due to early pruning in the parsing process, this diagnostic approach moves the field away from blind trial-and-error toward informed, linguistic-driven optimization.

Summary updated

BibTeX
@inproceedings{hara2009effective,
  author = {Tadayoshi Hara and Yusuke Miyao and Jun'ichi Tsujii},
  title = {{Effective Analysis of Causes and Inter-dependencies of Parsing Errors}},
  booktitle = {IWPT},
  year = {2009},
  url = {https://openreview.net/forum?id=KXovXslLe3M},
}
Figure