TL;DR
This paper investigates whether large language models (LLMs) inherently understand frame semantics, a linguistic theory that models meaning through conceptual frames. The authors show that LLMs can perform frame identification (determining the correct semantic frame for a word in context) with minimal prompting, and that fine-tuning significantly boosts performance to state-of-the-art levels, even generalizing to out-of-domain tasks. This suggests LLMs possess a latent understanding of frame semantics that can be leveraged with appropriate techniques.
What Problem Does This Paper Address?
Understanding the meaning of words in context, especially polysemous words with multiple senses, is a fundamental challenge in Natural Language Understanding (NLU). Frame semantics offers a structured way to model these meanings by defining "frames" that represent typical situations or events and the roles (frame elements) involved. Frame Semantic Parsing aims to automatically extract these structures from text, with "frame identification" being a crucial subtask: identifying the correct frame evoked by a target word. Traditional methods rely heavily on supervised learning and extensive annotated data. This paper explores whether modern LLMs, particularly instruction-tuned ones, can perform this task effectively with little to no task-specific training, by leveraging their potentially encoded latent knowledge of frame semantics.
Methods
The authors explore two primary methods for evaluating LLMs' frame semantic knowledge:
- Inference-Time Prompting: They use simple, lightweight prompts to elicit frame identification from LLMs without fine-tuning. Two prompt formats are tested:
- Simple Prompt: Presents the sentence, target word, and candidate frames with their definitions, asking the model to output the most appropriate frame name.
-
Direct-QA Prompt: Similar to the simple prompt but frames are labeled (A, B, C, etc.), and the model selects the label in a Question-Answering format. Both zero-shot and few-shot (with a small number of examples) settings are evaluated.
-
QA Fine-Tuning: The LLM is fine-tuned on a question-answering formulation of the frame identification task. Each training instance includes a sentence, target word, and candidate frames with definitions, and the model learns to select the correct frame.
Additionally, the paper probes the LLMs' understanding by asking them to generate frame definitions and evaluating their quality extrinsically on the frame identification task.
Datasets
The experiments primarily use the FrameNet 1.5 and FrameNet 1.7 lexical resources, which provide sentence-level annotations linking lexical units (word senses) to semantic frames.
For out-of-domain evaluation, the following datasets are used:
- YAGS (Yet Another General Semantic dataset): Derived from user-generated content, introducing domain shift and challenging lexical ambiguity.
- Artifacts: Consists of noun phrases annotated with FrameNet frames, focusing on the prototypical functions of physical objects and requiring a shift from sentence-level to phrase-level reasoning.
Key Insights and Ideas
- LLMs possess latent frame knowledge: Even without task-specific fine-tuning, LLMs can perform frame identification reasonably well using simple prompts, indicating they have internalized some understanding of frame semantics.
- Prompting is powerful: Lightweight prompting strategies, especially when including frame definitions and lexical unit (LU) definitions, are surprisingly effective.
- Fine-tuning unlocks state-of-the-art performance: Fine-tuning LLMs on frame identification tasks significantly boosts performance, reaching and even exceeding previous state-of-the-art results on FrameNet benchmarks.
- Generalization is strong: The fine-tuned models demonstrate robust generalization capabilities across different domains and input formats (sentences vs. phrases).
- LLMs can generate useful frame definitions: LLMs can produce definitions that, when used in place of gold definitions, still allow for comparable frame identification accuracy, suggesting they capture essential semantic content.
Key Results
- Prompting Performance: Zero-shot prompting with the "Simple" format achieved accuracies around 80-83% on FrameNet 1.5 and 1.7. Few-shot prompting offered marginal improvements.
- Fine-tuning Performance: Fine-tuning Llama-3.1-8B achieved accuracies of 91.7% on FN 1.5 and 91.9% on FN 1.7, matching or surpassing prior state-of-the-art systems.
- Out-of-Domain Performance: The fine-tuned model achieved 80.7% on YAGS and significantly improved performance on Artifacts (from 25.6% to 49.6%), demonstrating strong generalization.
- Definition Generation: Replacing gold FrameNet definitions with LLM-generated ones resulted in comparable accuracy on the Frame Identification task, highlighting the utility of these generated definitions.
- LU Definitions Matter: Incorporating Lexical Unit (LU) definitions in prompts significantly improved performance, especially in ambiguous cases, acting as strong semantic cues.
Limitations and Future Work
- Model Specificity: The study primarily focused on the Llama family of models. Extending this to other LLMs and exploring multilingual contexts or alternative frame ontologies is an open question.
- Task Scope: The research is confined to frame identification, leaving other components of frame semantic parsing (target identification, argument identification) unaddressed.
- Definition Evaluation: The evaluation of LLM-generated frame definitions was primarily extrinsic (on the frame identification task). A more rigorous human annotation-based evaluation could provide deeper insights.
- Prompting Strategies for Definitions: The definition generation used a zero-shot prompting style. Exploring diverse prompting strategies (e.g., structured scaffolds, exemplars, chain-of-thought) might further enhance definition quality.
- Definition Precision: While LLMs can generate coherent definitions, they sometimes lack the precision, conciseness, and role-structure sensitivity of curated FrameNet definitions. Future work could explore controlled definition generation techniques to enforce these qualities and hybrid approaches to support under-resourced frames.
Summary updated