Abstraction and Generalisation in Semantic Role Labels: PropBank, VerbNet or both?

Paola Merlo, Lonneke van der PlasACL 2009

TL;DR

This paper compares two major semantic role labeling (SRL) schemes, PropBank and VerbNet, to understand their strengths in capturing linguistic generalizations. They find that while PropBank is easier to learn due to its syntactic alignment, VerbNet offers richer, verb-specific information and better generalization to new instances. The authors conclude that both resources are complementary and can be used together for more robust NLP applications.

What Problem Does This Paper Address?

Understanding the meaning of a sentence is crucial for many Natural Language Processing (NLP) tasks, such as question answering and machine translation. A key component of this understanding is identifying the "who did what to whom" – the participants in an event and their roles. Semantic Role Labels (SRLs) are used to represent these grammatically relevant aspects of sentence meaning. However, there are different ways to define and annotate these roles. This paper focuses on comparing two prominent SRL schemes: PropBank and VerbNet. PropBank, widely adopted in NLP, is closely tied to syntactic structure. VerbNet, on the other hand, is a lexical resource that aims for more verb-specific semantic roles. The challenge is to determine which scheme better captures linguistic generalizations about semantic roles and how well they "grammaticalize" meaning, and whether one is superior or if they can be combined.

Methods

The authors employ a task-independent, general analysis of PropBank and VerbNet. They use information-theoretic measures (entropy, joint entropy, mutual information, symmetric uncertainty) to quantify the relationships between semantic role labels, verbs, and verb classes. Crucially, they avoid using syntactic parsing or features to ensure their analysis is not confounded by the syntactic alignment of PropBank. They also analyze the co-occurrence patterns of role sets and investigate how semantic roles correlate with grammatical functions.

Datasets

The primary dataset used is the SemLink 1.1 corpus, which provides mappings between PropBank and VerbNet annotations for the Penn Treebank Wall Street Journal (WSJ) portion. The analysis focuses on occurrences where both PropBank and VerbNet labels are available, representing approximately 45% of PropBank semantic roles. For the analysis of semantic roles and grammatical functions, they merge SemLink data with the CoNNL 2008 shared task data on syntactic and semantic dependency parsing.

Key Insights and Ideas

  • PropBank vs. VerbNet: A Tale of Two Philosophies: PropBank's labels are more abstract and align closely with syntactic structure, making them easier for NLP models to learn. VerbNet, however, provides finer-grained, verb-specific roles that carry more semantic information but are harder to learn without verb context.
  • Complementarity is Key: Neither system is definitively "better" across the board. PropBank excels at providing generalizable, high-level abstractions, while VerbNet offers detailed, verb-specific disambiguation. This suggests they are most powerful when used together.
  • VerbNet's Richer Information: VerbNet's labels are more strongly correlated with specific verbs and convey more information, as evidenced by higher entropy and mutual information scores when verb information is available.
  • PropBank's Syntactic Advantage: PropBank's close ties to syntax make it a better predictor of grammatical functions, which is advantageous for models that heavily rely on syntactic parsing.
  • Role Sets Reveal Abstraction: When looking at combinations of roles (role sets), PropBank's sets appear to be higher-level abstractions of VerbNet's more granular sets, reinforcing the idea of different levels of generalization.

Key Results

  • Error Rate Reduction: In a task-independent comparison, VerbNet showed a greater reduction in error rate than PropBank when verb information was available. However, PropBank performed better when verb information was absent, highlighting its robustness in such scenarios.
  • Information Content: VerbNet's distribution of labels has a higher entropy (2.06 bits) compared to PropBank (1.37 bits), indicating it conveys more information.
  • Verb Correlation: The mutual information between VerbNet labels and verbs (1.03 bits) was significantly higher than for PropBank (0.26 bits), confirming VerbNet's stronger verb-specific nature.
  • Syntactic Correlation: PropBank showed a stronger correlation with syntactic dependency labels (symmetric uncertainty of 0.49) compared to VerbNet (0.39), aligning with its design principles.
  • Animacy and Generality: An analysis of pronoun distributions suggested that VerbNet labels might capture basic components of meaning more generally, as they showed a significant effect of animacy, unlike PropBank.

Limitations and Future Work

  • Task-Specific Limitations: The authors acknowledge that previous studies showing PropBank's superiority were often task-specific (e.g., parsing-based tasks) and might have been influenced by PropBank's syntactic alignment. Their task-independent approach aims to mitigate this.
  • Incomplete SemLink Data: The analysis was limited to instances where both PropBank and VerbNet labels were available in SemLink, which was only about 45% of PropBank semantic roles.
  • Future Work: The authors suggest investigating the cross-linguistic validity of semantic role labeling schemes as a future direction. They also implicitly suggest that future research could explore how to effectively combine PropBank and VerbNet for improved NLP performance.

Summary updated

BibTeX
@inproceedings{merlo2009abstraction,
  author = {Paola Merlo and Lonneke van der Plas},
  title = {{Abstraction and Generalisation in Semantic Role Labels: PropBank, VerbNet or both?}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2009},
}
Figure