Using Bracketed Parses to evaluate a Grammar Checking Application

Richard H. Wojcik, Philip Harrison, John BremerACL 1993 · DOI

Introduction

Evaluating a grammar checker presents a unique challenge compared to standard natural language processing tasks. While a standard parser is judged by its ability to produce a linguistically "correct" tree, a grammar checker's primary goal is to provide helpful, accurate feedback to a writer. Researchers at Boeing addressed this by developing a methodology to evaluate the Boeing Simplified English Checker (BSEC). Their approach decouples internal structural performance from external error-reporting accuracy, revealing that even imperfect linguistic models can be highly effective tools for end users.

The Evaluation Gap

The authors identified a significant gap in traditional evaluation: a system might produce a linguistically incorrect parse tree but still trigger the correct error message (e.g., flagging a missing article). Conversely, a "perfect" parser might fail to catch an error if the underlying error-detection rules are flawed. To bridge this gap, the study sought to measure how well the parser understands structure versus how accurately it generates error critiques.

Methodology: The Three-Pronged Approach

The team introduced an evaluation strategy using a tool called spe (string, parse, and evaluate), which forces the parser to consider only trees matching a specific hand-written "gold-standard" bracketing template. They generated three distinct reports to isolate different variables of system performance:

  1. Unbracketed Report: The system running normally on raw text.
  2. Unmodified Bracketed Report: The system forced to use "correct" structures but limited by its existing grammar rules.
  3. Modified Bracketed Report: The system forced to use "correct" structures after researchers added missing vocabulary and rules to ensure the parser could handle the sentences.
Table 1: Comparison of checker outcomes across unbracketed and bracketed conditions.

By comparing these runs, the researchers calculated two key metrics for error critiques:

  • Precision: The percentage of reported errors that were legitimate.
  • Recall: The percentage of "real" errors (identified in the ideal bracketed run) that the normal system successfully caught.

Dataset and Context

The study utilized a corpus of 297 sentences sampled from approximately 20,000 sentences found in Boeing maintenance documents. These documents are written in Simplified English (SE), a controlled language used in the aerospace industry to maximize clarity and safety. The test sentences ranged from 3 to 32 words, with a median length of 12 words.

Key Insights: The Utility of "Weird" Parses

One of the most significant findings was the system's resilience. The BSEC often produces "senseless" parses—syntactically valid but semantically bizarre structures. However, because these "weird" parses often involve the same error-triggering rules as the "correct" parse, the user still receives the correct advice.

This is largely due to the BSEC's use of a full-ambiguity parser. By generating a "forest" of possible trees, the system is more likely to find at least one path that identifies a legitimate error than a system that commits to a single "best" parse early in the process.

Results and Performance Metrics

The evaluation revealed high levels of accuracy for the BSEC in its target domain:

  • System Coverage: The parser achieved an 84% success rate in producing the optimal parse.
  • Critique Precision: Approximately 79% of the error messages shown to users were correct.
  • Critique Recall: The system successfully caught 89% of the actual errors present in the text.
Table 2: Breakdown of error categories, comparing unbracketed reports against actual errors and spurious detections.

As shown in the data, the system identified 302 errors in the unbracketed run, of which 64 were spurious. Despite these false positives, the high recall rate for categories like Part of Speech (POS) and Non-Simplified English (NON-SE) words demonstrates the tool's practical utility in a technical writing environment.

Conclusion and Future Work

While the results are specific to the aerospace industry's Simplified English, the methodology provides a robust framework for diagnosing grammar checker performance. The study acted as a diagnostic tool, revealing "obscure bugs" where the error report generator failed to flag certain categories even when the parser identified them correctly. The authors plan to use this test corpus as a training set to refine the system before repeating the evaluation with a fresh corpus to measure improvements in precision and recall.

Summary updated

BibTeX
@inproceedings{wojcik1993using,
  author = {Richard H. Wojcik and Philip Harrison and John Bremer},
  title = {{Using Bracketed Parses to evaluate a Grammar Checking Application}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {1993},
  doi = {10.3115/981574.981580},
}
Figure