Exploiting a Verb Lexicon in Automatic Semantic Role Labelling

Robert Swier, Suzanne StevensonEMNLP 2005

TL;DR

This paper introduces an unsupervised semantic role labeling (SRL) system that leverages a computational verb lexicon (VerbNet) instead of relying on expensive hand-labeled corpora. By matching parsed sentence constituents to verb frames in VerbNet, the system generates candidate roles, which are then used to train a simple probability model to resolve ambiguities, achieving significant improvements over a baseline without requiring manual annotations.

What Problem Does This Paper Address?

Automatic Semantic Role Labeling (SRL) is a crucial step towards enabling computers to understand the meaning of sentences. It involves identifying the semantic roles (like Agent, Patient, or Location) that different parts of a sentence play with respect to a verb. Traditionally, SRL systems rely heavily on supervised learning, which requires large amounts of hand-labeled data (corpora where humans have manually assigned semantic roles to sentence constituents). Creating these corpora is extremely time-consuming and expensive, limiting the portability of SRL systems to new languages or domains. This paper tackles the challenge of building an effective SRL system without this dependency, exploring how to utilize readily available lexical resources like verb lexicons.

Methods

The core of this work is a novel approach that bypasses the need for hand-labeled corpora by exploiting VerbNet, a computational lexicon that details possible semantic roles for verbs and their syntactic frames.

  1. Frame Matching: The system first parses sentences and then uses a "frame matcher" to align extracted sentence constituents with potential verb frames in VerbNet for the target verb. This process identifies possible semantic roles for each constituent based on syntactic position (subject, object, etc.) and prepositional phrases.
  2. Candidate Role Initialization: The frame matching process is designed to be restrictive, aiming to assign a single, unambiguous role to as many constituents as possible. This "primary-labeled data" forms the initial, albeit noisy, training set.
  3. Probability Model: For constituents where the frame matcher identifies multiple possible roles (ambiguous cases), a simple probability model is trained on the primary-labeled data. This model learns to predict the most likely role based on the verb and the constituent's syntactic slot.
  4. Role Mapping: A significant methodological challenge was that VerbNet's roles are more general than those in FrameNet, a widely used hand-labeled corpus. To evaluate their system, the authors developed a three-step mapping process to convert FrameNet roles into VerbNet roles, creating a new, adapted evaluation corpus.

Datasets

  • VerbNet: Used as the primary lexical resource for its detailed verb frames and semantic role assignments.
  • FrameNet II: The primary source of sentences for development and testing. The authors developed a mapping to convert FrameNet's semantic roles to VerbNet roles for evaluation purposes.
  • British National Corpus (BNC): Used to augment the FrameNet data for experiments involving more training data.

Key Insights and Ideas

  • Lexicons as a Viable Alternative: The paper strongly demonstrates that verb lexicons, like VerbNet, can serve as a powerful resource for SRL, significantly reducing or even eliminating the need for hand-labeled corpora.
  • Frame Matching's Power: The restrictive nature of the frame matching process is key. It successfully resolves a large portion of role ambiguities on its own, highlighting the value of explicit lexical knowledge.
  • Bootstrapping Resources: The developed role mapping technique shows a practical way to leverage existing, well-established resources (like FrameNet) to create evaluation data for systems based on lexicons that may not have associated annotated corpora.
  • Simplicity Can Be Effective: The simplest probability models (P(r|sc) and P(r|s)) performed as well as or better than more complex ones, suggesting that with a strong initial candidate selection from the frame matcher, sophisticated models might not be necessary.

Key Results

  • The system achieved an F-measure of .83 on labelling correctly extracted arguments, a substantial improvement over an informed baseline of .74.
  • On the overall task (identification and labelling), the system achieved an F-measure of .65, outperforming the baseline of .52. While this is lower than state-of-the-art supervised systems, it's achieved without any hand-labeled training data.
  • The frame matcher alone achieved an F-measure of .60 on the combined task, indicating that much of the labeling work is done by the lexicon itself.
  • Using the "best frames" (syntactically closest matches) in the frame matcher yielded better results than considering "all frames," suggesting that precise candidate selection is beneficial.

Limitations and Future Work

  • Noisy Primary Data: The primary-labeled data generated by the frame matcher is inherently noisy, which can affect the training of the probability model.
  • Role Mapping Inconsistencies: The mapping from FrameNet to VerbNet can introduce inconsistencies, requiring adjustments (like converting certain roles to "NoRole") to ensure evaluation data aligns with VerbNet.
  • Performance Gap: The system's performance on the overall task (.61-.65 F-measure) is still below supervised systems (around .82), indicating room for improvement.
  • Future Directions: The authors suggest that extending the system with more sophisticated probabilistic features or incorporating human filtering/augmentation could further improve performance and potentially match supervised system levels, while still offering a significant advantage in terms of resource requirements.

Summary updated

BibTeX
@inproceedings{swier2005exploiting,
  author = {Robert Swier and Suzanne Stevenson},
  title = {{Exploiting a Verb Lexicon in Automatic Semantic Role Labelling}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2005},
}
Figure