Maximum Entropy Models for FrameNet Classification

Michael Fleischman, Namhee Kwon, Eduard H. HovyEMNLP 2003 · OpenReview

Abstract

The development of FrameNet, a large database of semantically annotated sentences, has primed research into statistical methods for semantic tagging. We advance previous work by adopting a Maximum Entropy approach and by using previous tag information to find the highest probability tag sequence for a given sentence. Further we examine the use of sentence level syntactic pattern features to increase performance. We analyze our strategy on both human annotated and automatically identified frame elements, and compare performance to previous work on identical test data. Experiments indicate a statistically significant improvement (p<0.01) of over 6%.

Introduction

Semantic Role Labeling (SRL) is a critical task in natural language understanding, focused on identifying "who did what to whom" in a sentence. This involves mapping constituents of a sentence to semantic roles defined within a specific frame. For instance, in the "Body-Movement" frame, a system must correctly identify the Agent, the Body Part, and the Cause of the movement.

Example of semantic role labeling for the "Body-Movement" frame.

The primary resource for this research is FrameNet, a linguistically rich database. However, FrameNet presents challenges due to data sparsity, with many predicates having very few annotated examples. Previous state-of-the-art models, such as the linear interpolation approach by Gildea and Jurafsky (2002), struggled to integrate diverse features effectively and required complex backoff strategies.

Methodology: Maximum Entropy Framework

To address these limitations, this study adopts a Maximum Entropy (ME) framework. ME models are log-linear, allowing for the seamless integration of a wide variety of features without the need for the rigid decomposition required by traditional probabilistic models. The model automatically learns the optimal weights for each feature, facilitating superior feature fusion.

Key technical innovations include:

  • Global Syntactic Patterns: The model incorporates the sequence of all phrase types and functions in a sentence, capturing the broader structural context.
  • Sequence Modeling with Viterbi Search: Recognizing that semantic roles are interdependent (e.g., a single verb rarely has multiple "Agents"), the authors utilized previous role tags as features and employed Viterbi decoding to find the most probable sequence of roles for an entire sentence.
  • Smoothing: Gaussian priors were used to prevent overfitting, which is particularly important given the sparse nature of the FrameNet training data.

Experimental Results

The models were evaluated using the June 2002 release of FrameNet, maintaining the exact training and test splits used in the Gildea and Jurafsky (2002) baseline to ensure a fair comparison.

The experiments demonstrated that the ME approach significantly outperforms the previous state-of-the-art. The most advanced model (Experiment 3), which included both global syntactic patterns and sequence modeling, achieved an accuracy of 84.7% on the classification of human-annotated frame elements. This represents a statistically significant improvement (p < 0.01) of 6.2% over the 78.5% baseline.

Comparison of classifier performance on the test set across different experimental setups.

Discussion and Future Work

While the classification of known frame elements has seen substantial improvement, the task of identification—finding the boundaries of these elements—remains a significant challenge. The authors found that while classification accuracy continues to rise with more data, identification performance (measured by F-score) appears to plateau.

Future research will focus on developing a unified tagging framework that integrates both identification and classification into a single Maximum Entropy model. Such a joint modeling approach could potentially overcome current performance bottlenecks and lead to even more robust semantic role labeling systems.

Summary updated

Abstract

The development of FrameNet, a large database of semantically annotated sentences, has primed research into statistical methods for semantic tagging. We advance previous work by adopting a Maximum Entropy approach and by using previous tag information to find the highest probability tag sequence for a given sentence. Further we examine the use of sentence level syntactic pattern features to increase performance. We analyze our strategy on both human annotated and automatically identified frame elements, and compare performance to previous work on identical test data. Experiments indicate a statistically significant improvement (p<0.01) of over 6%.
BibTeX
@inproceedings{fleischman2003maximum,
  author = {Michael Fleischman and Namhee Kwon and Eduard H. Hovy},
  title = {{Maximum Entropy Models for FrameNet Classification}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2003},
  url = {https://openreview.net/forum?id=H1WFLGGuZS},
}
Figure