jp-evalb: Robust Alignment-based PARSEVAL Measures

Jungyeul Park, Junrui Wang, Eunkyul Leah Jo, Angela Yoonseo ParkNAACL 2024 · DOI

TL;DR

This paper introduces jp-evalb, a new evaluation system for constituency parsing that addresses limitations of the standard evalb tool. By incorporating sentence and word alignment, jp-evalb offers a more robust and flexible evaluation, particularly for end-to-end systems where tokenization and sentence boundaries might differ between gold and system outputs.

What Problem Does This Paper Address?

Evaluating constituency parsers has traditionally relied on the evalb tool, which implements the PARSEVAL measures. While widely used, evalb imposes strict constraints: it requires identical tokenization and sentence boundaries between the gold standard and system-generated parse trees. This rigidity can lead to inaccurate evaluations, especially in modern end-to-end parsing pipelines where variations in preprocessing steps (like tokenization or sentence splitting) are common. These discrepancies can cause evalb to incorrectly flag valid parses as errors, hindering a true assessment of parser performance. The paper aims to overcome these issues by providing a more adaptive evaluation framework.

Methods

The core innovation of jp-evalb lies in its alignment-based approach. Instead of demanding exact matches, jp-evalb first aligns sentences and then words between the gold and system outputs. This alignment process allows the system to handle variations in tokenization and sentence boundaries.

The algorithm proceeds in two main stages:

  1. Sentence and Word Alignment: The system aligns sentences and then words using a pattern-matching approach. This is inspired by techniques used in machine translation for sentence and word alignment. The alignment considers factors like sentence length and word similarity, using conditions to match tokens and phrases.
  2. Constituency Evaluation: After alignment, jp-evalb calculates PARSEVAL measures (precision, recall, F-score) based on the aligned trees. It counts true positives, false positives, and false negatives for constituents. Crucially, jp-evalb also reports on word accuracy and POS tagging accuracy, including punctuation marks, which evalb often excludes.

The paper also provides a "legacy option" (-evalb) to reproduce the exact output of the original evalb, ensuring backward compatibility and allowing for direct comparison.

Datasets

The paper mentions using the English Penn Treebank for evaluation. They also present a case study on Korean end-to-end parsing evaluation using the Korean Sejong treebank. No new datasets were introduced.

Key Insights and Ideas

  • Alignment is Key for Robustness: The fundamental insight is that aligning sentences and words before evaluation is crucial for handling real-world variations in parsing pipelines, making the evaluation more forgiving and accurate.
  • Beyond Strict Matching: jp-evalb moves beyond the brittle exact-match requirement of evalb, acknowledging that minor differences in preprocessing shouldn't invalidate a parser's core performance.
  • Comprehensive Evaluation: By including punctuation in its analysis and reporting word/POS tag accuracy more comprehensively, jp-evalb offers a richer picture of parser performance than evalb.
  • Preserving Legacy: The -evalb option is a smart move, allowing researchers to transition gradually and verify that jp-evalb can indeed replicate evalb's behavior when needed.

Key Results

  • jp-evalb successfully handles sentence and word mismatches, providing accurate evaluations where evalb might fail or report errors/skips.
  • The paper demonstrates that jp-evalb can achieve perfect precision and recall even with word mismatches (e.g., "can't" vs. "can not") due to its alignment capabilities.
  • In a case study on the English Penn Treebank, jp-evalb required approximately 4.5 seconds for evaluation, compared to evalb's less than 0.1 seconds. This highlights a trade-off between robustness and speed, with jp-evalb being slower but more comprehensive.
  • The system correctly evaluates cases involving trace symbols (*-num) that cause evalb to fail.
  • For Korean parsing, jp-evalb successfully aligned sentences and morphemes despite deviations from gold standards, assessing constituency parsing performance accurately.

Limitations and Future Work

  • Runtime Overhead: The paper acknowledges that the sentence and word alignment process introduces additional runtime compared to evalb. While jp-evalb is not claimed to be faster, its robustness is presented as a worthwhile trade-off.
  • Future Work: The authors suggest that their alignment-based approach offers a more robust and reliable evaluation method, paving the way for more comprehensive assessments of constituency parsing, especially in complex, end-to-end systems. The paper implies that this approach can be integrated into existing PARSEVAL frameworks, ensuring consistency.

Summary updated

BibTeX
@inproceedings{park2024jp,
  author = {Jungyeul Park and Junrui Wang and Eunkyul Leah Jo and Angela Yoonseo Park},
  title = {{jp-evalb: Robust Alignment-based PARSEVAL Measures}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-demo.7},
}
Figure