Semantic Frame Induction using Masked Word Embeddings and Two-Step Clustering

Kosuke Yamada, Ryohei Sasano, Koichi TakedaACL 2021

TL;DR

This paper introduces a novel semantic frame induction method that leverages masked word embeddings and a two-step clustering approach. By masking frame-evoking verbs, the method reduces reliance on superficial word information, leading to more accurate frame assignments. The two-step clustering first groups instances of the same verb before clustering across verbs, resulting in a more appropriate number of semantic frames.

What Problem Does This Paper Address?

Semantic frame induction aims to automatically discover and group sentences that express the same underlying meaning (semantic frame) based on frame-evoking words, typically verbs. While recent methods using contextualized word embeddings like BERT have shown promise, they suffer from two main drawbacks:

  1. Over-reliance on superficial verb information: The embeddings often capture surface-level features of the verb itself, which can overshadow the actual semantic frame being conveyed. This can lead to instances of the same verb being incorrectly clustered if they have different surface forms or if different verbs are used to express the same frame.
  2. Excessive frame granularity: Current methods tend to cluster all instances across all verbs simultaneously. This can result in the same verb being split into too many distinct frame clusters, especially when some instances are outliers or when the clustering algorithm struggles to differentiate subtle semantic nuances.

The authors aim to overcome these limitations to build more comprehensive and accurate semantic frame resources.

Methods

The proposed method combines two key innovations:

  • Masked Word Embeddings: Instead of solely relying on standard contextualized word embeddings (e.g., from BERT), the authors also utilize masked word embeddings. This involves replacing the frame-evoking verb in a sentence with a special "[MASK]" token and then obtaining its embedding. The intuition is that masking the verb forces the model to rely more on the surrounding context to understand the meaning, thereby reducing the influence of the verb's superficial form. They also explore a weighted average of the standard and masked embeddings, allowing for a tunable balance between contextual and masked information.
  • Two-Step Clustering: This approach addresses the issue of over-segmentation.
  • First Step (Intra-verb Clustering): Instances of the same verb are clustered together first. This step focuses on grouping semantically similar instances of a single verb, effectively identifying "pseudo-lexical units" (pLUs). The authors found that masked embeddings are particularly effective here. They experimented with X-means and group average clustering.
  • Second Step (Inter-verb Clustering): The pLUs generated in the first step are then clustered across different verbs. This allows for the merging of semantically similar pLUs from different verbs into a single semantic frame. Ward clustering and group average clustering were used.

A novel termination criterion based on the ratio of pLU pairs belonging to the same cluster compared to the ratio of actual LU pairs belonging to the same frame in a development set is used to determine the number of clusters.

Datasets

The experiments were conducted using the English FrameNet data release 1.7.

  • Verbal LUs with at least 20 example sentences were extracted.
  • To manage the dataset size, a maximum of 100 example sentences per LU were used, with random selection if more were available.
  • The dataset was split into a development set (20% of verbs) and a test set (80% of verbs).
  • The development set was used to tune the hyperparameter alpha for the masked embeddings and the termination criterion for clustering.
  • The authors note that they did not use the SemEval-2019 Task 2 dataset as it was no longer available.

Key Insights and Ideas

  • Masking is key to disentangling verb form from meaning: The experiments clearly show that masked word embeddings significantly improve the separation of instances belonging to different frames, especially when compared to standard contextualized embeddings. This suggests that focusing on context rather than the verb's surface form is crucial for accurate frame induction.
  • Two-step clustering prevents over-segmentation: By first clustering within verbs and then across them, the method effectively groups semantically similar instances while avoiding the creation of too many granular clusters for a single verb.
  • A balanced approach to embeddings is beneficial: While masked embeddings are powerful, a weighted combination with standard embeddings (v_word+mask) can offer a tunable way to balance contextual information with the verb's inherent meaning. The optimal alpha value was found to be close to 1.0, indicating a strong preference for masked embeddings.
  • A principled termination criterion is essential: The proposed ratio-based criterion for determining the number of clusters proved effective, aligning well with the actual number of frames in the development set.

Key Results

  • The proposed two-step clustering method, particularly when using X-means for the first step and group average clustering for the second, achieved the highest B-CUBED F-SCORE (BCF) of 64.4 and the highest PIF score of 73.0 among the evaluated models.
  • These results outperformed previous state-of-the-art methods from SemEval-2019 Task 2, even though the dataset used in this study is considered more challenging due to a higher proportion of verbs evoking multiple frames.
  • The number of generated clusters (410) was close to the number of human-annotated frames (393), indicating the effectiveness of the termination criterion.
  • The study found that for the two-step clustering, a value of alpha close to 1.0 was optimal, suggesting that masked embeddings are more critical than standard embeddings for the inter-verb clustering step. In contrast, one-step clustering methods using v_word+mask with alpha=0.0 (effectively just standard embeddings) showed no benefit from masking.

Limitations and Future Work

  • Effectiveness of 1cpv' strategy: The paper notes that the 1cpv' strategy (treating all instances of a verb as a single cluster in the first step) performed surprisingly well on this specific dataset. However, they believe its effectiveness might be limited on more complex datasets where verbs have more distinct meanings. Further investigation is needed.
  • Focus on verbs: The current method primarily focuses on verbs as frame-evoking words. Future work aims to extend the approach to include nouns and adjectives.
  • Argument induction and role labeling: The ultimate goal is to build a comprehensive frame-semantic resource, which requires not only frame induction but also the determination of arguments and their semantic roles.
  • Utilizing argument embeddings: A potential extension is to incorporate contextualized word embeddings of arguments of verbs to see if this can generalize the approach for argument role induction.

Summary updated

BibTeX
@inproceedings{yamada2021semantic,
  author = {Kosuke Yamada and Ryohei Sasano and Koichi Takeda},
  title = {{Semantic Frame Induction using Masked Word Embeddings and Two-Step Clustering}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure