Towards Robust Semantic Role Labeling

Sameer Pradhan, Wayne Ward, James MartinNAACL 2007

TL;DR

This paper investigates the robustness of Semantic Role Labeling (SRL) systems when applied to text from different genres than they were trained on. They found that while identifying which words are arguments is relatively robust, classifying their semantic roles degrades significantly, suggesting that current SRL models overfit to specific lexical and semantic features of their training data.

What Problem Does This Paper Address?

Semantic Role Labeling (SRL) is a crucial task in Natural Language Processing (NLP) that aims to identify and label the semantic arguments of a predicate (usually a verb) in a sentence. This is vital for applications like Information Extraction and Question Answering. Most existing SRL research focuses on training and evaluating systems on the same corpus, often the Wall Street Journal (WSJ) dataset. While this leads to high performance on WSJ, the authors hypothesize that this strategy results in systems that are "over-fit" to the specific genre of the WSJ. This over-fitting poses a significant problem for the widespread adoption and utility of SRL technology, as real-world applications often involve text from diverse sources and genres. The paper aims to quantify this performance degradation across genres and understand its underlying causes.

Methods

The authors employ a Support Vector Machine (SVM) classifier for the SRL task, a common approach in recent SRL research. They formulate the problem as a classification task, breaking it down into three sub-tasks:

  • Argument Identification (ID): Determining which constituents in a sentence represent a semantic role for a given predicate.
  • Argument Classification (Class.): Assigning the correct semantic role label (e.g., ARGO, ARG1) to a pre-identified constituent.
  • ID + Class.: Performing both identification and classification.

They compare performance using both hand-corrected Treebank parses and automatically generated parses from the Charniak parser. A key aspect of their methodology is the cross-genre evaluation, specifically training on the WSJ corpus and testing on the Brown corpus, and vice-versa. They also conduct experiments to isolate the impact of syntactic parser errors versus inherent SRL classification challenges.

Datasets

  • PropBanked WSJ: The Wall Street Journal corpus, annotated with semantic roles based on PropBank. Used for training and testing.
  • PropBanked Brown Corpus: A portion of the Brown Corpus, also annotated with semantic roles based on PropBank. Used for testing and, in some experiments, for training. The Brown Corpus is noted as being a more diverse collection of English text compared to the WSJ.
  • Penn Treebank: Used for syntactic parsing.

Key Insights and Ideas

  • Genre Matters Significantly for Classification: The most striking finding is the substantial drop in performance for argument classification when moving between genres (WSJ to Brown, or vice-versa). Argument identification, however, is much more robust.
  • Syntactic Parser Errors are Not the Primary Culprit: While syntactic parser errors do contribute to performance degradation, the experiments suggest they are not the main reason for the large drop in classification accuracy. Even with gold-standard parses, classification performance suffers across genres.
  • Lexical/Semantic Features Dominate Classification: The authors posit that the classification task relies heavily on features that are specific to the training genre's vocabulary and semantic nuances. These features are likely "over-fit" and don't generalize well.
  • Structural Features Drive Identification: In contrast, argument identification seems to be driven by more general structural features of the parse tree (like the path from the argument to the predicate), which are more consistent across genres.
  • Data Size vs. Genre: While more training data generally helps, the genre of the data appears to be a more critical factor for classification performance than simply the amount of data.

Key Results

  • When an SRL system trained on WSJ is tested on the Brown corpus, the F-score for ID+Class. drops from 91.2% (WSJ test) to 79.9% (Brown test).
  • The degradation for Argument Identification (ID) is much smaller, dropping from 96.8% to 85.5% (using hand-corrected parses).
  • Even with gold-standard syntactic parses, the performance gap between WSJ and Brown for classification remains significant.
  • Adding data from a new genre (Brown) to a WSJ-trained system gradually improves performance, with about 75% of the new data accounting for most of the performance gain.

Limitations and Future Work

  • Overfitting to Specific Features: The paper clearly identifies overfitting to genre-specific lexical and semantic features as a major limitation of current SRL systems.
  • Future Work Suggestions: The authors suggest two potential remedies:
  • Using less homogeneous corpora: Training on a more diverse set of genres from the outset.
  • Using less specific features: Replacing highly specific features (like proper names) with more general ones (like named entity types) to encourage the learning of more generalizable patterns. They acknowledge that these approaches might reduce performance on the original training genre but should improve cross-genre generalization.
  • Verb Sense Information: While verb sense information shows a small improvement, it doesn't fully address the genre transfer problem, suggesting other factors are more dominant.

Summary updated

BibTeX
@inproceedings{pradhan2007towards,
  author = {Sameer Pradhan and Wayne Ward and James Martin},
  title = {{Towards Robust Semantic Role Labeling}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2007},
}
Figure