Parser Evaluation Across Frameworks without Format Conversion

Wai Lok Tam, Yo Sato, Yusuke Miyao, Junichi TsujiiCF+CDPE @ COLING 2008 · OpenReview

Abstract

Evaluating natural language parsers built on different grammar frameworks usually requires messy, error-prone format conversions to compare their outputs. This paper proposes a novel, conversion-free evaluation method: extracting the names of specific linguistic phenomena (like "passive voice" or "raising") directly from a parser's native output and comparing them against a phenomenon-based gold standard.

The Challenge of Cross-Framework Evaluation

To evaluate how well a "deep parser" understands a sentence, researchers typically compare its output to a human-annotated gold standard. However, the NLP community uses various grammar frameworks (like HPSG, CCG, or Dynamic Syntax), and each outputs data in wildly different formats—from phrase structure trees to complex semantic graphs.

To compare parsers across these frameworks, the standard approach has been to convert their diverse outputs into a simplified, "universal" format, like Grammatical Relations (GR) or Stanford Dependencies (SD). The problem is that format conversion is incredibly lossy and difficult. Even if a parser is 90% accurate, the script converting its output to a universal format might only succeed 80% of the time. This makes it impossible to tell if a bad score is due to a failing parser or a failing format converter. The authors aim to evaluate parsers fairly across frameworks without forcing them into a single, unnatural output format.

Methods

Instead of comparing syntax trees or dependency graphs, the authors propose evaluating parsers based on linguistic phenomena.

  1. Creating a Phenomenon-Based Gold Standard: Instead of drawing full syntax trees, human annotators tag a test suite of sentences with the linguistic phenomena they contain. To avoid annotation fatigue, they use a non-exhaustive approach: they tag each sentence with its core phenomenon (e.g., "unshifted ditransitive") and the most common error a parser is likely to make.
  2. Native Phenomena Recognition: For each parser being evaluated, the developers write a lightweight "recognizer" script. This script reads the parser's native output and flags which phenomena are present. Because this script is written by someone intimately familiar with the parser's specific framework, it avoids the pitfalls of cross-framework translation.
  3. Custom Precision and Recall: The authors redefine Precision and Recall using set theory. These custom formulas compare the recognizer's exhaustive list of found phenomena against the annotator's non-exhaustive "core phenomenon + common error" list to generate a fair score.

Dataset and Test Suite

The authors created a custom, small test suite of 10 sentences. Each sentence was specifically chosen to illustrate core or complex linguistic phenomena, such as dative-shifted ditransitives, nominal gerunds, and inversion.

The 10-sentence test suite used to evaluate linguistic phenomena.

Key Results

The custom recognizer scripts achieved a 100% success rate in identifying the targeted phenomena from the parsers' native outputs. This proves that extracting phenomena natively is vastly more reliable than cross-framework format conversion. The authors successfully evaluated two distinct deep parsers against their 10-sentence gold standard:

  • ENJU (an HPSG-based parser) achieved 0.8 Precision and 0.7 Recall.
  • DSPD (a Dynamic Syntax-based parser) achieved 0.7 Precision and 0.5 Recall.
Precision and Recall metrics for the ENJU and DSPD parsers.

Limitations and Future Work

The authors are transparent about the limitations of this early-stage proposal. The experiment was conducted on a micro-scale of only 10 sentences, serving as a proof-of-concept rather than a definitive benchmark. Additionally, the custom precision metric can be overly harsh; if a parser gets a complex sentence mostly right but misses one specific role assignment, a strict recognizer might give it a score of 0 for that phenomenon, ignoring partial correctness.

Future work will involve scaling up the test data to determine whether exhaustive or non-exhaustive annotation is ultimately better for large-scale, cross-framework evaluations.

Summary updated

Abstract

Wai Lok Tam, Yo Sato, Yusuke Miyao, Junichi Tsujii. Coling 2008: Proceedings of the workshop on Cross-Framework and Cross-Domain Parser Evaluation. 2008.
BibTeX
@inproceedings{tam2008parser,
  author = {Wai Lok Tam and Yo Sato and Yusuke Miyao and Junichi Tsujii},
  title = {{Parser Evaluation Across Frameworks without Format Conversion}},
  booktitle = {CF+CDPE @ COLING},
  year = {2008},
  url = {https://openreview.net/forum?id=LH2SEPVUCX-},
}
Figure