The Role of Lexicalization and Pruning for Base Noun Phrase Grammars

Claire Cardie, David R. PierceAAAI 1999 · OpenReview

Abstract

This paper explores the role of lexicalization and pruning of grammars for base noun phrase identification. We modify our original framework (Cardie & Pierce 1998) to extract lexicalized treebank grammars that assign a score to each potential noun phrase based upon both the part-of-speech tag sequence and the word sequence of the phrase. We evaluate the modified framework on the "simple" and "complex" base NP corpora of the original study. As expected, we find that lexicalization dramatically improves the performance of the unpruned treebank grammars; however, for the simple base noun phrase data set, the lexicalized grammar performs below the corresponding unlexicalized but pruned grammar, suggesting that lexicalization is not critical for recognizing very simple, relatively unambiguous constituents. Somewhat surprisingly, we also find that error-driven pruning improves the performance of the probabilistic, lexicalized base noun phrase grammars by up to 1.0% recall and 0.4% precision, and does so even using the original pruning strategy that fails to distinguish the effects of lexicalization. This result may have implications for many probabilistic grammar-based approaches to problems in natural language processing: error-driven pruning is a remarkably robust method for improving the performance of probabilistic and non-probabilistic grammars alike.

Introduction

Identifying "base noun phrases" (NPs)—simple, non-recursive phrases like "[the survival]" or "[spinoff Cray Computer Corp.]"—is a foundational task in Natural Language Processing (NLP). These phrases serve as the building blocks for information extraction, indexing, and partial parsing. Earlier work by Cardie and Pierce (1998) utilized a simple grammar based only on Part-of-Speech (POS) tags. To refine this grammar, they employed "error-driven pruning" to remove rules that frequently led to misidentifications. However, the rise of lexicalized parsers raised a critical question: does adding word-level information make error-driven pruning obsolete? This study investigates the interplay between lexicalization and pruning in the context of base NP identification.

Methodology

The authors extended the original "Treebank Approach" to incorporate lexical information and sophisticated decoding. The process is divided into two main phases: grammar extraction and noun phrase identification.

Overview of the base noun phrase identification framework.

Lexicalized Scoring

Instead of a binary check for POS sequences, the system assigns a score $S(b)$ to every potential NP bracket. This score is a linear combination of three factors:

  1. The probability of the POS tag sequence.
  2. The probability of the first and last words appearing within that specific tag sequence.
  3. A "length score" that favors longer phrases, effectively mimicking the "longest-match" heuristic commonly used in NLP.

Graph-Based Decoding

To determine the optimal set of NPs for a sentence, all possible bracketings are represented as a weighted graph. Dynamic programming is then used to find the path through the graph with the highest total score, ensuring a globally optimal bracketing for the sentence.

Error-Driven Pruning

Two pruning strategies were evaluated for the lexicalized grammar:

  • Thresholding: Discarding rules that fail to meet a minimum "net benefit" score on a validation set.
  • Incremental: Iteratively discarding the $N$ worst-performing rules to refine the grammar.

Experimental Setup

The study utilized data from the Penn Treebank (Wall Street Journal), categorized into two datasets:

  • Complex Base NPs: The standard definition of non-recursive noun phrases.
  • Simple Base NPs: A simplified version where ambiguities like conjunctions and prepositions within the NP were removed to create a "cleaner" baseline. Evaluation was conducted using 5-fold cross-validation across 25 sections of the corpus.

Results and Discussion

The experiments revealed several key insights into the necessity of both lexicalization and pruning.

Performance comparison of lexicalized and unlexicalized grammars.

The Impact of Lexicalization and Pruning

As shown in Figure 5, lexicalization provides a massive performance boost for unpruned grammars, increasing precision and recall by approximately 70 and 55 percentage points, respectively, on the complex corpus. However, pruning remains essential. For unlexicalized grammars, pruning is the primary driver of accuracy. Even for lexicalized models, incremental pruning improved performance by up to 1.0% recall and 0.4% precision. This suggests that error-driven pruning is a robust method that complements probabilistic scoring by explicitly removing problematic rules.

Heuristics and Context

The "longest-match" heuristic proved surprisingly powerful, often contributing more to the accuracy of unpruned models than specific word data. Furthermore, for the "Simple" corpus, a pruned POS-only grammar actually outperformed the more complex lexicalized version (92.1% P / 93.1% R vs 91.5% P / 92.6% R). This indicates that lexicalization is most critical when dealing with high linguistic ambiguity. Finally, the authors noted that their model, while effective, still trailed behind Transformation-Based Learning (TBL). They attribute this to TBL's ability to consider the surrounding context (words outside the NP), which appears to be more informative than the lexical content within the NP itself.

Conclusion

This research demonstrates that lexicalization and error-driven pruning are not mutually exclusive but rather complementary techniques. While lexicalization significantly enhances the performance of base NP grammars, pruning remains a vital tool for optimizing both probabilistic and non-probabilistic models. Future work will explore more formal probabilistic frameworks, such as HMMs or PCFGs, and investigate methods to incorporate surrounding context into the scoring function to further improve identification accuracy.

Summary updated

Abstract

This paper explores the role of lexicalization and pruning of grammars for base noun phrase identification. We modify our original framework (Cardie & Pierce 1998) to extract lexicalized treebank grammars that assign a score to each potential noun phrase based upon both the part-of-speech tag sequence and the word sequence of the phrase. We evaluate the modified framework on the "simple" and "complex" base NP corpora of the original study. As expected, we find that lexicalization dramatically improves the performance of the unpruned treebank grammars; however, for the simple base noun phrase data set, the lexicalized grammar performs below the corresponding unlexicalized but pruned grammar, suggesting that lexicalization is not critical for recognizing very simple, relatively unambiguous constituents. Somewhat surprisingly, we also find that error-driven pruning improves the performance of the probabilistic, lexicalized base noun phrase grammars by up to 1.0% recall and 0.4% precision, and does so even using the original pruning strategy that fails to distinguish the effects of lexicalization. This result may have implications for many probabilistic grammar-based approaches to problems in natural language processing: error-driven pruning is a remarkably robust method for improving the performance of probabilistic and non-probabilistic grammars alike.
BibTeX
@inproceedings{cardie1999role,
  author = {Claire Cardie and David R. Pierce},
  title = {{The Role of Lexicalization and Pruning for Base Noun Phrase Grammars}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {1999},
  url = {https://openreview.net/forum?id=ryZE0TxdZr},
}
Figure