Introduction
In natural language processing, the evaluation of parsers is critical for measuring progress. Historically, the "PARSEVAL" metric, which counts phrase boundaries (brackets), has been the standard. However, this approach often fails to reflect the true semantic accuracy of a parse. This paper proposes a dependency-based evaluation method that focuses on the relationships between heads and modifiers, offering a more intuitive and flexible alternative.
The Limitations of PARSEVAL
The traditional bracketing approach suffers from several significant drawbacks:
- Sensitivity to Minor Errors: A single attachment mistake can disrupt multiple nested brackets, leading to disproportionately low scores.
- The "Flat Tree" Loophole: Parsers that produce minimal structure can achieve high precision scores without providing useful information.
- Linguistic Disagreements: Different theories may dictate different attachment points for certain words (e.g., adverbs). PARSEVAL penalizes these stylistic differences even when the core meaning is preserved.
Proposed Dependency-based Framework
The proposed method represents sentences as a set of dependency triples: (modifier, head, relationship). For instance, in the phrase "big dog," big is the modifier of the head dog. The primary metric is the error count, defined as the number of words assigned an incorrect head.
To facilitate this, the paper introduces an algorithm (makeDeps) and a "Tree Head Table" to convert existing constituency-based treebanks into dependency structures. This ensures the method can be integrated into current research workflows.
The "Modify" Module and Normalization
A key innovation is the "Modify" module, which allows researchers to normalize trees before evaluation. By applying operations such as Deletion, Inversion, and Transfer, different grammatical conventions can be aligned. This ensures that a parser is evaluated on its ability to capture essential relationships rather than its adherence to a specific, potentially arbitrary, annotation style.
Evaluation and Results
The dependency-based metric provides greater diagnostic power. By isolating specific relations, researchers can pinpoint exactly where a parser fails. In complex cases, such as prepositional phrase attachments, the metric accurately reflects that a single attachment error constitutes a single mistake.
As shown in the comparison above, the dependency framework clearly identifies errors (marked "yes") in the parser's output for complex sentences like "I saw a man with a dog and a cat in the park." This granularity is missing in traditional bracketing metrics.
Conclusion
By shifting the focus from phrase boundaries to dependency relations, this method provides a more semantically relevant and robust evaluation for broad-coverage parsers. While it requires the standardization of head-finding rules, it offers a significant improvement over the PARSEVAL standard in terms of both intuition and diagnostic utility.
Summary updated