TL;DR
This paper tackles the limited coverage of frame-to-frame relations in FrameNet, a crucial resource for semantic analysis. They introduce a supervised machine learning approach that leverages structural features from FrameNet, semantic relations from WordNet, and corpus-based lexical associations to predict new, potentially missing relations. Their method significantly outperforms baselines and shows promise for enriching semantic knowledge bases.
What Problem Does This Paper Address?
Frame semantics, a theory of meaning, is foundational for understanding natural language. FrameNet is a prominent lexical semantic resource built upon this theory, providing a network of semantic frames and their relationships. However, a significant challenge with FrameNet is its incomplete coverage of these frame-to-frame relations. While other work has explored unsupervised methods to address this, they often lack rigorous evaluation against FrameNet's own structure. This paper aims to bridge this gap by developing a supervised approach to systematically enrich FrameNet's relational structure, making it a more comprehensive resource for semantic analysis tasks like information extraction and question answering.
Methods
The authors propose a supervised machine learning framework to predict frame-to-frame relations. Their approach is two-fold: first, they train a binary classifier to distinguish between existing FrameNet relations and randomly sampled "unrelated" frame pairs. Second, they apply this trained model to predict new relations for frame pairs not currently listed in FrameNet.
The core of their methodology lies in the rich set of features used to train the classifier. These features are drawn from three main sources:
- WordNet-Based Features: This leverages the hierarchical and relational structure of WordNet. They project WordNet's synset (set of synonyms) relations (like hypernymy/hyponymy) onto FrameNet frames by considering the lexical units (words that evoke frames) within each frame. They use both occurrence-based features (counting relation existence) and similarity-based features (calculating semantic similarity between synsets using measures like Path, Wu-Palmer, and Leacock-Chodorow).
- FrameNet Based Features: This utilizes the existing structure of FrameNet itself. Features include frame element overlap (shared argument roles between frames) and a measure inspired by Hirst-St.Onge's semantic similarity, adapted to FrameNet's relation network. They also incorporate a definition overlap feature, comparing the open-class words in frame definitions.
- Corpus-Based Features: This draws information from large text corpora.
- Frame Co-occurrences: They compute point-wise mutual information (PMI) of frames co-occurring in the GigaWord corpus. To handle frame ambiguity (where a word can evoke multiple frames), they use a weighted co-occurrence measure based on probabilities derived from FrameNet.
- Lexical Co-occurrences: They extract semantic associations between verbs based on their co-occurrences with discourse markers, categorized into six types of relations (causal, temporal, continuation, contrast, disjunction, elaboration). These are then mapped to potential FrameNet relations.
For training, they use a Random Forest classifier, known for its robustness. They also employ a balanced dataset of positive and negative examples to address class imbalance.
Datasets
- FrameNet: Used for its existing frame relations, frame definitions, and lexical units. It also served as the basis for deriving frame co-occurrence probabilities.
- WordNet: Used for its lexical and semantic relations between synsets, projected onto FrameNet frames.
- GigaWord Corpus: A large-scale corpus used for computing frame co-occurrences.
- Penn Discourse Treebank (PDTB): Used to extract semantic associations between verbs based on discourse markers.
- SemCor Corpus: Used for learning word-sense probabilities (though the paper notes their approach is slightly different).
The paper creates a balanced dataset of 1648 frame-pairs (824 positive, 824 negative) for training and evaluation. A subset of this is used for testing. They also generate a set of "distractors" (randomly sampled frame pairs absent from FrameNet) for human evaluation.
Key Insights and Ideas
- Synergy of Features: The paper strongly suggests that combining features from different sources (WordNet, FrameNet structure, and corpus data) is crucial for effectively enriching FrameNet relations. No single feature group alone achieves the performance of the full model.
- FrameNet's Bias: The authors acknowledge that FrameNet-based features perform best, likely because the existing FrameNet network is denser in areas that have been the focus of lexicographic work. This highlights a potential bias in the resource itself, but also shows the power of leveraging existing structured knowledge.
- Human Agreement: The study demonstrates that human annotators can achieve acceptable inter-annotator agreement (Cohen's Kappa of 0.65) when judging the validity of predicted frame relations, even with limited information. This validates the feasibility of human evaluation for this task.
- Completeness of FrameNet: The error analysis reveals that some predicted relations might represent genuinely missing relations or inconsistencies in FrameNet's current structure, particularly concerning antonymous events or finer-grained distinctions.
Key Results
- Accuracy Improvement: The "Full Model" (combining all feature groups) achieved a test accuracy of 88.4%, significantly outperforming baselines based on individual feature groups (WordNet-based: 63.4%, Frame information: 79.3%, Corpus-based: 60.4%).
- Precision in New Relation Prediction: When predicting new relations and applying a confidence cutoff of 0.8, the model achieved a precision of 63% on a manually evaluated set of 100 predicted pairs.
- High Proportion of Potential Relations: Out of the top 100 positive candidates predicted by the model, 63 were judged as positive by human annotators. This suggests that a substantial number of frame pairs could potentially be related, even if not explicitly defined in FrameNet.
- Distractor Robustness: Only 4 out of 100 "distractor" pairs (randomly sampled unrelated pairs) were judged positive by annotators, indicating the model is relatively good at distinguishing true relations from random associations.
Limitations and Future Work
- Recall Estimation: The paper acknowledges the difficulty in estimating the recall of their model, as the true number of existing relations is unknown.
- Imbalanced Data: While they balanced the training data, the inherent sparsity of frame relations means the problem is still imbalanced, and they note that this might lead to a degradation of precision on new instances, which they control with a confidence threshold.
- Error Analysis Insights: The error analysis points to potential issues with FrameNet's completeness and consistency, particularly regarding antonymous events and the granularity of frames.
- Future Work:
- Global Relation Prediction: Explore predicting relations in a more global way, considering existing and predicted relations to avoid redundant or conflicting links.
- Lexical Expansion Integration: Combine their approach with work on automatically expanding FrameNet's lexicon.
- FE-to-FE Relation Linking: Explore automatically linking frame elements (arguments) for the newly proposed frame-to-frame relations.
- Pre-filtering Candidates: Use looser lexical association resources to pre-filter candidate pairs before applying the supervised model.
- Confidence Levels: Evaluate model predictions at various confidence levels to find a better compromise between precision and recall for human validation.
Summary updated