Can Semantic Roles Generalize Across Genres?

Szu-ting Yi, Edward Loper, Martha PalmerNAACL 2007

TL;DR

This paper tackles the problem of Semantic Role Labeling (SRL) systems overfitting to specific text genres, like the Wall Street Journal (WSJ), leading to poor performance on new genres. They propose a solution by mapping PropBank's verb-specific role labels to VerbNet's more general, consistent thematic roles, demonstrating improved generalization and robustness for SRL systems.

What Problem Does This Paper Address?

Semantic Role Labeling (SRL) is a crucial task in Natural Language Processing (NLP) that aims to identify the constituents of a sentence that act as arguments to a verb and assign them appropriate semantic roles (e.g., who did what to whom). PropBank is a widely used corpus for training SRL systems, but it has a significant limitation: its role labels are defined on a per-verb basis. This means that the same label (e.g., Arg2) can represent very different semantic roles for different verbs (e.g., "material" for "make" vs. "extent" for "multiply"). This verb-specific nature leads to models that overfit to the idiosyncrasies of the training data, typically the WSJ corpus. Consequently, these models perform poorly when applied to text from different genres (like the Brown corpus), hindering the development of robust and generalizable NLP applications. The paper aims to bridge this gap by creating more consistent and verb-independent role labels.

Methods

The core of the paper's contribution lies in creating a mapping between PropBank and VerbNet. VerbNet is a lexical resource that organizes verbs into classes with hierarchically arranged thematic roles, designed for consistency across verbs. The mapping process involves two main steps:

  1. Lexical Mapping: This step identifies potential mappings between PropBank labels and VerbNet thematic roles for a given word.
  2. Instance Classifier: This component, trained on a semi-automatically annotated corpus, determines the most appropriate VerbNet mapping for a specific occurrence of a word in context, essentially performing word sense disambiguation for role labels.

To address the issue of sparse data when mapping to individual VerbNet roles (which increased the number of labels from 6 to 23), the authors grouped similar VerbNet thematic roles into five coherent categories. This resulted in a new tag set: Arg0 (agent), Arg1 (patient), Group1 (goal), Group2 (extent), Group3 (predicate/attrib), Group4 (product), and Group5 (instrument/cause). The authors then re-trained their existing SRL system using these new, more generalized labels.

Datasets

  • PropBank: Used as the primary source of annotated data for semantic role labeling. The training data is primarily from the Wall Street Journal (WSJ) portion of the Penn Treebank II.
  • VerbNet: A lexical resource used to provide more consistent, verb-independent thematic roles.
  • Penn Treebank II: Used for training and testing the SRL system. Sections 02-21 were used for training, and section 23 for testing on the same genre (WSJ).
  • Brown Corpus: Used as a test set to evaluate the system's performance on a different genre than the training data. The PropBank-ed portion of the Brown corpus was used.

Key Insights and Ideas

  • Verb-Specific vs. Verb-Independent Roles: The paper strongly highlights the limitations of verb-specific role labels (PropBank) and the benefits of verb-independent thematic roles (VerbNet) for generalization.
  • Overloading of Arg2-5: The analysis clearly shows that PropBank's Arg2-5 labels are significantly more overloaded and inconsistent across verbs compared to Arg0 and Arg1.
  • Grouping Thematic Roles is Crucial: Directly mapping to individual VerbNet roles led to data sparsity. Grouping similar roles was a necessary step to make the mapping practical and effective.
  • Robustness Gains: The proposed mapping significantly improves SRL system robustness, enabling better performance on unseen genres.
  • Arg1's Verb-Independence: The results confirm that Arg1 is indeed relatively verb-independent, and mapping it to VerbNet roles provides minimal improvement, while Arg2 benefits substantially from the mapping due to its verb-dependent nature.

Key Results

  • Arg2 Improvement: The mapping of Arg2 to VerbNet thematic role groups resulted in a 6% performance improvement on the WSJ corpus and a 10% improvement on the Brown corpus.
  • Arg1 Performance: For Arg1, the performance was statistically similar between the original and mapped systems, confirming its verb-independent nature. The precision increase was offset by a recall decrease due to data sparseness.
  • Improved Distinction for Arg2: The mapping made Arg2 more clearly defined. In instances where the original system failed, the mapped system correctly identified 233 instances that were previously unlabeled and resolved 1,252 instances where the original system confused Arg2 with other labels.
  • Overall Performance: While direct comparison is tricky due to different tag sets, the mapped system showed comparable or improved performance, especially on the cross-genre Brown corpus, indicating better generalization.

Limitations and Future Work

  • Data Sparseness for Arg3-5: The authors acknowledge that the sample size for mapped Arg3-5 instances is very small, which poses challenges for building robust classifiers for these arguments.
  • Limited Testing on Novel Genres: While the Brown corpus was used, more extensive testing on diverse genres is needed to fully evaluate robustness.
  • Future Work:
    • Map the PropBank-ed Brown corpus to VerbNet for more thorough testing.
  • Explore "back-off" techniques to handle out-of-vocabulary verbs using verb class membership.
  • Investigate the effect of different thematic role groupings on system performance.

Summary updated

BibTeX
@inproceedings{yi2007can,
  author = {Szu-ting Yi and Edward Loper and Martha Palmer},
  title = {{Can Semantic Roles Generalize Across Genres?}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2007},
}
Figure