Fully Parsing the Penn Treebank

Ryan Gabbard, Seth Kulick, Mitchell MarcusNAACL 2006

TL;DR

This paper introduces a two-stage parser that fully reconstructs Penn Treebank-style syntactic analyses, including function tags and empty categories, which are often omitted by standard parsers. The first stage integrates function tag recovery with minimal code changes to an existing parser, while the second stage leverages machine learning to achieve state-of-the-art performance in recovering empty categories.

What Problem Does This Paper Address?

Modern statistical parsers, while adept at generating skeletal syntactic structures, often discard valuable linguistic information present in the Penn Treebank. Specifically, they ignore "function tags" (e.g., -SBJ for subject, -TMP for temporal) that clarify syntactic and semantic roles, and "empty categories" (e.g., null complementizers, traces of wh-movement) that represent displaced constituents crucial for understanding predicate-argument structure. While previous work has tackled these individually, this paper aims to recover both types of information simultaneously, producing a richer, more complete syntactic representation that is closer to the original Treebank annotations. This is important because richer representations are essential for downstream tasks like semantic role labeling and can significantly reduce the manual effort required for creating new treebanks.

Methods

The paper proposes a two-stage parsing approach:

  • Stage 1: Function Tag Recovery: This stage involves a minimal modification of the Collins parser (specifically, Bikel's implementation of Model 2). The core idea is to retain the function tags that the parser already uses internally for argument identification, rather than deleting them in a post-processing step. This is achieved by modifying less than ten lines of code, allowing the parser to output function tags without sacrificing its standard Parseval metric performance. The argument identification heuristic is also slightly augmented to treat all syntactic group tags as arguments.
  • Stage 2: Empty Category Recovery: This stage employs a cascade of maximum-entropy and perceptron-based classifiers to recover empty categories. Crucially, this stage leverages the function tags recovered in the first stage as features, along with other linguistic features. The approach is inspired by rule-based systems like Campbell's but implemented using machine learning, aiming to combine the strengths of both. It addresses different types of empty categories (null complementizers, wh-traces, and null NPs) through specialized classifiers.

Datasets

  • Penn Treebank: Used for training and evaluation. Specifically, sections 2-21 were used for training, and section 23 for testing. Section 24 was used for development testing.

Key Insights and Ideas

  • Minimal Modification, Maximal Gain: The paper demonstrates that integrating function tag recovery into an existing parser requires surprisingly few code changes, yet significantly enriches the output. This suggests that many powerful parsers already possess the underlying information, which is simply discarded.
  • Function Tags as Crucial Features: The success of the second stage in recovering empty categories is heavily attributed to the availability of function tags from the first stage. This highlights the interconnectedness of different linguistic annotations and the power of using richer input features for machine learning models.
  • Post-processing vs. Integration: The work shows that a sophisticated post-processing approach for empty category recovery, informed by integrated function tags, can outperform purely integrated systems that might suffer from data sparsity issues.
  • Bridging Rule-Based and Machine Learning: The second stage effectively combines the linguistic insights of rule-based systems with the learning power of modern machine learning, achieving state-of-the-art results.

Key Results

  • Function Tagging: The first-stage parser achieves near state-of-the-art performance on recovering function tags, with results comparable to or exceeding previous specialized systems, despite minimal modifications. The Parseval labelled recall/precision scores for the modified parser are 88.23/88.31, virtually identical to the unmodified parser.
  • Empty Category Recovery: The two-stage system outperforms other state-of-the-art post-processing systems for empty category recovery. For example, it reduces error by 21% on (NP )s compared to the best PSLB post-processing approach and by 41% on wh-traces over Levy and Manning. It also outperforms Campbell's rule-based system across all categories, with significant error reductions on 0s (87%), (ADVP T)s (11%), (NP T)s (7%), and (NP )s (8%).

Limitations and Future Work

  • Evaluation Metrics: The authors acknowledge the limitations of standard metrics like Parseval, which ignore function tags and empty categories, and Johnson's metric, which can fail to catch attachment errors. They note that while other metrics exist, they also have weaknesses, making direct comparison challenging.
  • Data Fragmentation: The enriched space of node labels due to function tags initially posed a challenge for the Collins statistical model, but the authors managed to mitigate this.
  • Future Work: The authors envision their system providing rich syntactic guidance for recovering predicate-argument structure in the future. They also expect their tool to significantly enhance the productivity of syntactic annotation for new genres of English. They hope to have practical evidence of this in the near future.

Summary updated

BibTeX
@inproceedings{gabbard2006fully,
  author = {Ryan Gabbard and Seth Kulick and Mitchell Marcus},
  title = {{Fully Parsing the Penn Treebank}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2006},
}
Figure