Descriptive and Empirical Approaches to Capturing Underlying Dependencies among Parsing Errors

Tadayoshi Hara, Yusuke Miyao, Jun'ichi TsujiiEMNLP 2009 · OpenReview

Tadayoshi Hara, Yusuke Miyao, Jun’ichi Tsujii
Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing. 2009.

Introduction: Debugging the Parser

When a natural language parser fails, it rarely fails just once. Because language is a deeply interconnected web of dependencies, a single mistake—such as misidentifying the antecedent of a relative clause—can trigger a cascade of "downstream" errors. If a researcher only looks at the raw error count, they might conclude that the parser has numerous independent bugs, when in reality, it may have one root problem and several symptoms.

In this 2009 EMNLP paper, researchers from the University of Tokyo and the University of Manchester propose a systematic way to move past "symptom-level" analysis. They introduce two complementary approaches to capture the underlying dependencies between parsing errors, helping developers focus their efforts on the mistakes that actually matter.

The Problem: Shallow Error Analysis

Modern deep parsers (like HPSG or CCG parsers) are typically evaluated on predicate-argument relations—essentially, "who did what to whom." If a parser incorrectly identifies a relation, it generates an error.

Comparison between a correct parse and a parser output showing a mismatch in prepositional attachment.

The problem is that traditional error analysis is shallow. If a parser makes multiple mistakes in one sentence, a researcher usually has to manually inspect the output to see if those mistakes are related. This is slow, subjective, and difficult to scale.

A complex sentence where multiple errors (dotted lines) occur, illustrating the difficulty of manual error dependency analysis.

The authors aimed to automate the discovery of these "error dependencies" to reveal which linguistic phenomena (like prepositional phrase attachment or coordination) are the true culprits behind system failure.

Methodology: Descriptive and Empirical Approaches

The authors propose two distinct strategies to identify root causes:

1. The Descriptive Approach

This is a top-down method. The researchers defined 14 specific linguistic "error patterns" based on expert knowledge. For example, if a preposition’s argument is wrong, it’s flagged as a "Prepositional Attachment" error. By matching these patterns against the parser's output, they can group related errors into known linguistic categories.

2. The Empirical Approach

This is a bottom-up, "what-if" simulation. The researchers take a sentence where the parser failed and forcibly correct one specific error. They then re-parse the sentence under this new constraint. If other errors suddenly disappear, those errors are deemed "dependent" on the one that was corrected. This allows the system to find dependencies that a human might not have anticipated or written a pattern for.

By combining these, they can distinguish between independent errors (the root cause) and induced errors (the symptoms).

Evaluation and Key Insights

The researchers evaluated their methods using the Enju parser (a high-performance HPSG parser) on the Penn Treebank (PTB).

  • The "Correction Effect": One of the most powerful metrics introduced is the "correction effect"—the total number of errors fixed by resolving a single root cause. This helps prioritize which bugs a developer should fix first.
  • Independence vs. Symptom: The study found that some error types are almost always "independent" (like prepositional attachment), meaning they are the primary source of trouble. Others, like "Head selection for noun phrases," were almost never independent; they were almost always symptoms of a larger structural collapse.
  • The Power of Re-parsing: The empirical approach is particularly clever because it uses the parser’s own internal logic to reveal dependencies. It doesn't require the researcher to understand why the errors are related; the parser's search space provides the answer.

Key Results

  • Coverage: Using just 14 descriptive patterns, the authors were able to account for over 40% of all parsing errors in the test set.
  • Clustering: The empirical approach successfully grouped 3,085 correctable errors into 1,978 co-occurring groups.
  • The "Big Fish": Prepositional attachment was identified as the most significant independent error. Fixing these errors had a massive "multiplier effect," often resolving multiple other predicate-argument mismatches simultaneously.
  • Inductive Relations: They found 501 "one-way" dependencies where Error A always causes Error B, but Error B doesn't necessarily cause Error A.

Limitations and Future Work

The empirical approach relies on the parser being able to produce the correct answer if forced. However, if the correct structure was "pruned" (discarded) by the parser early on due to efficiency settings, the empirical method cannot analyze it. Additionally, the descriptive approach is limited by the human-written patterns, currently missing about 60% of errors. Future work involves expanding the pattern library and developing ways to handle "uncorrectable" errors where the gold-standard structure is outside the parser's search space.

Summary updated

BibTeX
@inproceedings{hara2009descriptive,
  author = {Tadayoshi Hara and Yusuke Miyao and Jun'ichi Tsujii},
  title = {{Descriptive and Empirical Approaches to Capturing Underlying Dependencies among Parsing Errors}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2009},
  url = {https://openreview.net/forum?id=7hcIp-DrjHi},
}
Figure