Accurate Parsing of the Proposition Bank

Gabriele Musillo, Paola MerloNAACL 2006

TL;DR

This paper presents a novel approach to jointly parse sentence structure and identify semantic roles, achieving state-of-the-art results on both tasks. By integrating semantic role information directly into the parsing process, they demonstrate that syntactic and semantic understanding can mutually benefit each other, leading to richer and more accurate linguistic representations.

What Problem Does This Paper Address?

Understanding the meaning of a sentence goes beyond just its grammatical structure. We need to know who did what to whom, where, and when. This is the domain of semantic role labeling (SRL). While statistical parsers have become very good at identifying the syntactic structure of sentences (like identifying noun phrases, verb phrases, etc.), applying similar techniques to SRL has proven challenging. Previous attempts to incorporate semantic information into parsing, or to use parsing results to improve SRL, have often led to a degradation in parsing accuracy. This paper tackles the problem of building a single, integrated system that can accurately perform both syntactic parsing and semantic role labeling simultaneously, without sacrificing performance on either task.

Methods

The authors leverage a family of state-of-the-art history-based statistical parsers called Simple Synchrony Network (SSN) parsers. These parsers use a left-corner parsing strategy and a novel neural network architecture to learn rich representations of the parsing history. The key innovation here is the augmentation of the parser's input features. Instead of just using standard syntactic labels, they incorporate PropBank semantic role labels directly into the parser's feature set. This is achieved by creating new, augmented part-of-speech tags that combine PTB labels with PropBank semantic roles (e.g., PP-AM-TMP). This allows the parser to learn from semantic information at a very early stage of the parsing process. They also introduce binary features to help the SSN generalize across different types of semantic role labels.

Datasets

  • Penn Treebank (PTB): Used as the base for syntactic parsing.
  • PropBank: A corpus that annotates verbal predicates with semantic roles. The authors use sections 2-21 for training, section 24 for validation, and section 23 (from the CoNLL-2005 shared task) for testing.

Key Insights and Ideas

  • Synergy between Syntax and Semantics: The paper strongly suggests that syntactic structure and semantic roles are not independent. Knowledge about semantic roles can inform syntactic parsing, and vice-versa, leading to a mutually beneficial relationship.
  • Integrated Systems are Feasible: Building a single system that handles both tasks is not only possible but can outperform separate systems, especially when the tasks are complex.
  • Early Semantic Information is Crucial: Incorporating semantic role information as early as possible in the parsing process is key to achieving good results.
  • Robustness to Complexity: The proposed model can handle a much richer set of labels (combining syntactic and semantic information) without a significant drop in performance, even with limited training data.

Key Results

  • PTB Parsing Accuracy: The "SSN+Roles" model achieves an F-measure of 89.0% on the standard PTB parsing task, which is comparable to the original SSN parser (89.1%) and only slightly lower than the best CoNLL 2005 SRL systems (83.3-84.1%). This demonstrates that adding semantic role information does not degrade syntactic parsing performance.
  • PropBank Parsing Accuracy: The model achieves an F-measure of 82.8% on the PropBank parsing task. This is competitive with the performance of state-of-the-art semantic role labeling systems, even though their approach is more integrated.
  • Robustness: The model is robust, meaning it can be extended to a richer set of labels without needing more training data.

Limitations and Future Work

  • Direct Comparison to SRL Systems: The authors acknowledge that a direct, apples-to-apples comparison with existing semantic role labeling systems is not yet possible because their current work focuses on jointly parsing and labeling, rather than solely predicting semantic roles from pre-parsed trees. They state that current work aims to extend their parser to recover argument structure for each verb to enable this direct comparison.
  • Necessity of Parsing for SRL: While their results show that joint parsing and SRL can be performed successfully, they do not yet definitively prove that parsing is necessary for SRL. Previous work has shown that sophisticated chunking can achieve state-of-the-art SRL results without full parsing. Their work, however, provides the first direct test of a joint solution.

Summary updated

BibTeX
@inproceedings{musillo2006accurate,
  author = {Gabriele Musillo and Paola Merlo},
  title = {{Accurate Parsing of the Proposition Bank}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2006},
}
Figure