Verb Sense Clustering using Contextualized Word Representations for Semantic Frame Induction

Kosuke Yamada, Ryohei Sasano, Koichi TakedaACL 2021

TL;DR

This paper investigates how well modern contextualized word representations (like BERT) can distinguish between different meanings (frames) of the same verb. They found that BERT and its variants are highly effective, significantly outperforming older models, and also explored methods for automatically estimating the number of frames a verb might evoke.

What Problem Does This Paper Address?

Manually creating comprehensive semantic frame resources, which define the roles arguments play for a given verb (e.g., "who did what to whom"), is a labor-intensive process. While contextualized word embeddings like ELMo and BERT have shown great promise in understanding word meaning in context, it's not fully clear how well they capture the nuances of semantic frames. Specifically, verbs can evoke different frames depending on the context (e.g., "support" can mean physical support or evidential support). This paper aims to bridge this gap by evaluating how effectively these advanced representations can automatically distinguish these verb-specific frames, a crucial step towards unsupervised semantic frame induction.

Methods

The core of the methodology involves using contextualized word representations to cluster sentences based on the verb's meaning.

  1. Acquire Representations: For target verbs that evoke multiple frames, the authors extract contextualized word embeddings for each instance of the verb in its surrounding sentence. They experiment with seven different models: ELMO, BERT (base and large), RoBERTa, ALBERT, GPT-2, and XLNet.
  2. Clustering: These embeddings are then clustered using a Gaussian Mixture Model (GMM). The number of clusters is pre-set to match the known number of frames for that verb in the dataset.
  3. Evaluation: The resulting clusters are compared against the human-annotated frames using a "match rate" metric. They also investigate estimating the number of frames using information criteria like BIC and an adjusted BIC (a-BIC).

Datasets

The study utilizes two prominent English frame-semantic resources:

  • FrameNet: A widely used resource based on frame semantics theory, containing frames, lexical units, and annotated sentences.
  • PropBank: Developed for supervised learning, it defines framesets for verbs with semantic role labels.

The authors created specific datasets by selecting verbs that evoke at least two frames with a minimum of 20 annotated sentences. They used a maximum of 100 annotated sentences per frame for their experiments.

Key Insights and Ideas

  • BERT's Superiority: Deep bidirectional Transformer models like BERT, RoBERTa, XLNet, and ALBERT significantly outperform unidirectional models (GPT-2) and older contextual models (ELMO) in distinguishing verb frames. This highlights the power of bidirectional context for capturing subtle semantic differences.
  • Context is King: The paper strongly suggests that contextualized word representations are crucial for semantic frame induction, as they can capture the nuances that static word embeddings or simple predicate-argument structures miss.
  • Hierarchical Relations Matter (and Complicate): Frames with hierarchical relationships (common in FrameNet) are harder to distinguish. This suggests that while these relationships are linguistically meaningful, they can pose challenges for automated systems aiming to separate distinct frames.
  • Estimating Frame Count: The adjusted BIC (a-BIC) shows promise in automatically estimating the number of frames a verb evokes, achieving over 50% accuracy. This is a vital step for unsupervised frame induction, as it removes the need for prior knowledge of frame counts.

Key Results

  • Frame Distinction: BERTLARGE and RoBERTa achieved the highest macro-average match rates (around 0.79) for distinguishing verb frames across both FrameNet and PropBank datasets. BERTBASE, XLNet, and ALBERT also performed very well (around 0.75-0.76).
  • Model Performance: ELMO and GPT-2 showed significantly lower performance, indicating that the architecture and pre-training strategy of deep bidirectional Transformers are key to success.
  • Frame Count Estimation: Using the adjusted-BIC, the authors achieved Spearman's rank correlation coefficients of 0.177 (FrameNet) and 0.631 (PropBank) and accuracy over 0.5 for predicting the number of frames. The standard BIC tended to overestimate the number of frames.

Limitations and Future Work

  • Hierarchical Frames: The paper acknowledges that distinguishing between frames with hierarchical relations is challenging. Future work could explore how to better handle these relationships.
  • Verb-Centric vs. Frame-Centric: The current study focuses on distinguishing frames for individual verbs. Future work should investigate how to capture frames that are shared across multiple verbs, which is characteristic of resources like FrameNet.
  • Argument Role Labeling: This paper focuses on frame induction (identifying which frame is evoked). Future research should also leverage contextualized representations for identifying and grouping arguments by their semantic roles within those frames.
  • Other POS: The study is limited to verbs. Extending the approach to other parts of speech (like nouns) that can evoke frames is a natural next step.
  • Cross-Lingual and Domain Adaptation: The authors suggest that contextualized representations could be valuable for low-resource languages and specific domains due to their ability to learn from large, unannotated corpora. This remains an important avenue for future exploration.

Summary updated

BibTeX
@inproceedings{yamada2021verb,
  author = {Kosuke Yamada and Ryohei Sasano and Koichi Takeda},
  title = {{Verb Sense Clustering using Contextualized Word Representations for Semantic Frame Induction}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure