A Dependency-based Method for Evaluating Broad-Coverage Parsers

Dekang LinIJCAI 1995

Introduction

In natural language processing, the evaluation of parsers is critical for measuring progress. Historically, the "PARSEVAL" metric, which counts phrase boundaries (brackets), has been the standard. However, this approach often fails to reflect the true semantic accuracy of a parse. This paper proposes a dependency-based evaluation method that focuses on the relationships between heads and modifiers, offering a more intuitive and flexible alternative.

The Limitations of PARSEVAL

The traditional bracketing approach suffers from several significant drawbacks:

  1. Sensitivity to Minor Errors: A single attachment mistake can disrupt multiple nested brackets, leading to disproportionately low scores.
  2. The "Flat Tree" Loophole: Parsers that produce minimal structure can achieve high precision scores without providing useful information.
  3. Linguistic Disagreements: Different theories may dictate different attachment points for certain words (e.g., adverbs). PARSEVAL penalizes these stylistic differences even when the core meaning is preserved.

Proposed Dependency-based Framework

The proposed method represents sentences as a set of dependency triples: (modifier, head, relationship). For instance, in the phrase "big dog," big is the modifier of the head dog. The primary metric is the error count, defined as the number of words assigned an incorrect head.

The process of parser evaluation, illustrating the conversion from constituency trees to dependencies and the subsequent modification and evaluation steps.

To facilitate this, the paper introduces an algorithm (makeDeps) and a "Tree Head Table" to convert existing constituency-based treebanks into dependency structures. This ensures the method can be integrated into current research workflows.

The "Modify" Module and Normalization

A key innovation is the "Modify" module, which allows researchers to normalize trees before evaluation. By applying operations such as Deletion, Inversion, and Transfer, different grammatical conventions can be aligned. This ensures that a parser is evaluated on its ability to capture essential relationships rather than its adherence to a specific, potentially arbitrary, annotation style.

Evaluation and Results

The dependency-based metric provides greater diagnostic power. By isolating specific relations, researchers can pinpoint exactly where a parser fails. In complex cases, such as prepositional phrase attachments, the metric accurately reflects that a single attachment error constitutes a single mistake.

A comparison between a gold standard (Key) and a parser output (Answer) for a complex sentence, highlighting specific dependency errors.

As shown in the comparison above, the dependency framework clearly identifies errors (marked "yes") in the parser's output for complex sentences like "I saw a man with a dog and a cat in the park." This granularity is missing in traditional bracketing metrics.

Conclusion

By shifting the focus from phrase boundaries to dependency relations, this method provides a more semantically relevant and robust evaluation for broad-coverage parsers. While it requires the standardization of head-finding rules, it offers a significant improvement over the PARSEVAL standard in terms of both intuition and diagnostic utility.

Summary updated

BibTeX
@inproceedings{lin1995dependency,
  author = {Dekang Lin},
  title = {{A Dependency-based Method for Evaluating Broad-Coverage Parsers}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {1995},
}
Figure