Shallow Semantic Parsing using Support Vector Machines

Sameer S. Pradhan, Wayne H. Ward, Kadri Hacioglu, James H. Martin, Daniel JurafskyNAACL 2004 · OpenReview

Abstract

This paper presents a high-performance system for shallow semantic parsing—the task of identifying "who did what to whom"—by replacing traditional statistical classifiers with Support Vector Machines (SVMs). By combining a two-stage classification strategy with a rich set of new linguistic features, the authors achieved state-of-the-art results on the PropBank corpus. The study demonstrates that SVMs significantly outperform traditional decision tree models and explores the critical role of syntactic features in semantic role labeling.

Introduction

In Natural Language Processing (NLP), understanding the "meaning" of a sentence requires identifying the semantic roles associated with a verb (the predicate). For example, in the sentence "The police arrested the suspect," a system must correctly label "The police" as the Agent and "the suspect" as the Patient. This task, known as Shallow Semantic Parsing or Semantic Role Labeling (SRL), is complicated by data sparsity, extreme class imbalance (where most syntactic nodes are not arguments), and linguistic ambiguity.

Figure 1: Example of a syntax tree annotated with semantic roles such as ARG0 (Talker) and ARGM-TMP (Temporal modifier).

Methodology

The authors developed a system that shifts from simple statistical models to SVMs with polynomial kernels. To manage the multi-class nature of semantic roles, they utilized a "One vs. All" (OVA) classification strategy.

Two-Stage Architecture

To handle the vast number of "NULL" nodes (nodes that do not represent semantic arguments), the system employs a two-stage filter. First, a binary classifier identifies potential argument nodes. Then, a multi-class classifier assigns specific semantic labels only to those candidates. This architecture significantly optimizes training and processing time without compromising accuracy.

Feature Engineering

Beyond standard features like the syntactic "Path" and "Head Word," the researchers introduced several novel linguistic features to improve performance:

  • Verb Clustering: Groups semantically similar verbs to handle unseen predicates.
  • Partial Path: A generalized version of the syntactic path designed to reduce data sparsity.
  • Named Entities: Identifies whether a phrase refers to a person, organization, or location.
Figure 2: Table showing the incremental performance gains in classification accuracy and argument identification (Precision, Recall, F1) as various linguistic features are added.

Key Insights and Results

The system was evaluated on the PropBank corpus, achieving an F1 score of 86.7% on hand-corrected parses. Several key findings emerged from the experiments:

  • Algorithm Superiority: Using the same feature set, the SVM model achieved 88% accuracy, outperforming decision tree classifiers (79%) by a substantial margin.
  • The "Path" Paradox: The syntactic "Path" feature (the route from the verb to the argument in the tree) proved to be the most critical feature for identifying arguments but was among the least helpful for classifying their specific roles.
  • Dependency on Syntax: Performance is highly sensitive to the quality of the underlying syntactic parse. When using automatically generated parses, the F1 score dropped from 86.7% to 79.4%.
  • Generalization Challenges: When tested on the AQUAINT corpus (moving from financial news to general news), performance dropped to 63.3% F1, highlighting the difficulty of domain portability for lexical features.

Conclusion

By leveraging the power of SVMs and sophisticated feature engineering, this work established a new state-of-the-art for shallow semantic parsing. However, the reliance on high-quality syntactic trees and the drop in performance across different domains suggest that future work should focus on more robust generalization techniques and tighter integration between syntactic and semantic processing.

Summary updated

BibTeX
@inproceedings{pradhan2004shallow,
  author = {Sameer S. Pradhan and Wayne H. Ward and Kadri Hacioglu and James H. Martin and Daniel Jurafsky},
  title = {{Shallow Semantic Parsing using Support Vector Machines}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2004},
  url = {https://openreview.net/forum?id=B1ZxAzZ_-S},
}
Figure