Abstract
Syntactic parsers trained on treebanks often inherit "ad hoc" rules—structures resulting from annotation errors, typos, or dataset-specific quirks. This paper presents two novel methods for identifying these unreliable rules using structural similarity and bigram frequency. Evaluation on the Penn Treebank and Brown Corpus demonstrates that these methods can identify non-generalizing rules with over 96% accuracy, facilitating the development of more robust natural language processing systems that generalize better across different linguistic domains.
Introduction: The Problem of "Leaky" Grammars
When training a natural language parser, researchers extract a grammar—a set of syntactic rules—from a manually annotated treebank. However, these treebanks are rarely perfect. They frequently contain "ad hoc" rules: structures that appear due to human error, idiosyncratic document quirks, or inconsistencies in annotation.
A significant challenge is that frequency alone is a poor filter. While many erroneous rules are rare, many valid linguistic structures are also rare (the "long tail" of language). Conversely, some frequent rules may represent systematic annotation errors. If a parser learns these ad hoc rules, it becomes "leaky," potentially accepting ungrammatical input or failing to generalize when applied to new genres, such as moving from news text to fiction.
Methodology: Scoring Rule Reliability
The author proposes two distinct scoring methods to determine the reliability of a rule without relying solely on raw frequency:
- Whole Daughters Scoring (Similarity-based): This method seeks "positive evidence" by grouping rules into equivalence classes (e.g., treating singular and plural nouns as equivalent). It uses a modified Levenshtein distance to identify "highly similar" rules. A rule receives a high reliability score if it has many exact matches or close neighbors in the data. Rules that stand alone without similar counterparts are flagged as ad hoc.
- Bigram Scoring (Component-based): This method looks for "negative evidence" by decomposing complex rules into constituent bigrams of daughter nodes. The reliability of the entire rule is determined by its "weakest link"—the least frequent bigram it contains. A rule containing a sequence of parts that rarely appears elsewhere is likely an error or an ad hoc construction.
Key Results and Cross-Genre Performance
Both methods proved highly effective at identifying rules that fail to appear in held-out data. Manual inspection of 108 low-scoring rules revealed that over 60% were clearly ad hoc, consisting of outright errors (50.93%) or fragmented/non-English text.
As shown in the results, at specific thresholds, the methods achieved a 96.93% accuracy rate in flagging rules that did not appear in the development set. This predictive power is crucial for pruning grammars to improve generalization.
The importance of identifying generalizable rules is further underscored by cross-genre testing. When moving from the Wall Street Journal (WSJ) portion of the Penn Treebank to the more diverse Brown Corpus, the coverage of known rule types dropped significantly.
While the WSJ development set showed approximately 80% coverage for rule types, this figure plummeted to roughly 38% for the Brown Corpus. This drop highlights the fragility of parsers that overfit to specific treebank quirks and the necessity of the proposed reliability filters.
Discussion and Limitations
Despite their success, the methods face challenges with naturally diverse structures. Coordinate structures (e.g., lists) and complex clausal rules (S, SBAR) are often flagged as unreliable due to their inherent structural variety, even when correct. Additionally, the methods currently require some manual effort to define head-equivalent categories for different treebanks.
Future work aims to develop more corpus-independent methods and better disentangle the effects of raw frequency from structural similarity, moving toward truly robust, domain-agnostic parsing grammars.
Summary updated