FrameEOL: Semantic Frame Induction using Causal Language Models

Chihiro Yano, Kosuke Yamada, Hayato Tsukagoshi, Ryohei Sasano, Koichi TakedaEMNLP 2025

TL;DR

This paper introduces FrameEOL, a novel method for semantic frame induction that leverages causal language models (CLMs) like GPT and Llama. FrameEOL achieves state-of-the-art performance on English and Japanese FrameNet datasets, outperforming previous methods that relied on masked language models (MLMs), especially in low-resource scenarios.

What Problem Does This Paper Address?

Semantic frame induction is the task of grouping words that evoke similar meanings (semantic frames) based on their context. This is crucial for understanding language, as it helps disambiguate word meanings. While recent advances using masked language models (MLMs) like BERT have shown promise, they often require extensive fine-tuning. Causal language models (CLMs), known for their strong language comprehension and generation capabilities, have been largely unexplored for this task. The paper addresses this gap by proposing a CLM-based approach that can achieve high performance with less data and computational cost, particularly for languages with limited annotated resources like Japanese.

Methods

The core of the proposed method, FrameEOL, is a prompt-based approach designed to elicit frame embeddings from CLMs.

  • FrameEOL Prompting: FrameEOL crafts a specific prompt for CLMs that asks for the FrameNet frame evoked by a given verb in a particular sentence. The embedding of the final token in the CLM's output is used as the frame embedding. This differs from previous prompt-based methods that focused on sentence embeddings.
  • In-Context Learning (ICL): To improve performance, especially in low-resource settings, FrameEOL utilizes ICL. This involves providing the CLM with a few examples (demonstrations) of frame induction tasks within the prompt itself. This allows the model to adapt to the task without explicit fine-tuning.
  • Deep Metric Learning (DML): For enhanced performance, especially when supervised data is available, the paper incorporates DML. This technique fine-tunes the model to bring embeddings of words evoking the same frame closer together in the embedding space, while pushing apart embeddings of words evoking different frames. The authors use a triplet loss function and a parameter-efficient fine-tuning method called LoRA to manage computational costs.
  • Clustering: After obtaining frame embeddings, the paper employs standard clustering techniques (one-step and two-step clustering) to group the frame-evoking words into semantic frames.

Datasets

The experiments were conducted using the following datasets:

  • English FrameNet: The standard FrameNet resource for English was used for evaluation. The dataset was split into training, development, and test sets, with a focus on verbs as frame-evoking words.
  • Japanese FrameNet: A significantly smaller dataset for Japanese was used to evaluate the method's performance in low-resource scenarios. This dataset contains only about 3.2% of the examples found in English FrameNet.

Key Insights and Ideas

  • CLMs are powerful for frame induction: The paper demonstrates that CLMs, when prompted correctly, can effectively perform semantic frame induction, rivaling and even surpassing MLM-based methods.
  • ICL is a game-changer for low-resource languages: The effectiveness of ICL with CLMs is highlighted, showing that even with a small number of examples (around 20), performance comparable to fine-tuned MLMs can be achieved for Japanese. This opens up possibilities for semantic frame induction in languages with scarce resources.
  • Prompt engineering matters: The specific design of the FrameEOL prompt is crucial for eliciting frame-specific embeddings from CLMs.
  • DML further boosts performance: When sufficient data is available, DML significantly improves the performance of CLM-based frame induction, achieving state-of-the-art results.

Key Results

  • English FrameNet: FrameEOL with DML achieved the highest performance, with Gemma 3 and Llama 3.1 models reaching BcF scores of 71.9 and 70.8 respectively in the one-step clustering setting, surpassing the best MLM-based method's score of 67.9.
  • Japanese FrameNet: The CLM-based methods demonstrated superior performance compared to MLM-based approaches. The one-step clustering setting using LLM-jp-3 achieved a BcF score of 61.3, outperforming the MLM-based method's score of 60.0.
  • ICL Performance: In the no-tuning setting, CLM-based methods performed worse than MLMs. However, with at least 20 ICL examples, CLMs achieved slightly better performance than MLM-based methods without fine-tuning.

Limitations and Future Work

  • Generalizability: The experiments were limited to English and Japanese FrameNet. Further research is needed to assess the approach's generalizability to other languages and corpora.
  • Computational Requirements: The use of large CLMs entails significant computational resources, which could be a barrier to practical deployment in resource-constrained environments.
  • Prompt Leakage: While the paper investigates the effect of removing the term "FrameNet" from the prompt, suggesting it doesn't significantly impact performance, the potential for implicit knowledge leakage from pre-training remains a consideration.
  • Future Work: The authors suggest exploring the method's applicability to other languages and investigating more parameter-efficient fine-tuning techniques. They also highlight the potential of combining CLMs with ICL for effective semantic frame induction in low-resource scenarios.

Summary updated

BibTeX
@inproceedings{yano2025frameeol,
  author = {Chihiro Yano and Kosuke Yamada and Hayato Tsukagoshi and Ryohei Sasano and Koichi Takeda},
  title = {{FrameEOL: Semantic Frame Induction using Causal Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure