Abstract
The meaning of polysemous words often varies in a highly productive yet predictable way. Generalizing the regularity between conventional senses to derive novel word meaning is crucial for automated processing of non-literal language uses such as figurative expressions. We introduce a novel task called systematic word meta-sense extension (SWORME) to test and improve language models' ability to extend word meaning to denote new semantic domains (also called meta-senses) that bear regular semantic relations with existing senses. We found that language models prefer incremental lexical semantic change toward conceptually similar meta-senses such as logical metonymy, and are much worse at predicting highly non-literal meaning extensions such as metaphors. We propose a novel analogy-based method of word meaning extension, and show that it effectively improves language model systematicity in making both gradual and radical types of meta-sense extension. We further demonstrate that learning systematic meta-sense extensions benefits language models on multiple benchmarks of figurative language understanding.
Introduction: The Challenge of Figurative Language
Human language is incredibly flexible. We take words from one domain (the "meta-sense") and productively apply them to another. For instance, if we understand that "grasp" and "get" can describe both physical objects and abstract ideas, we can easily interpret a novel expression like "to steal an idea," even if we have only ever encountered "steal" in the context of physical theft. This ability to generalize patterns across semantic domains is a hallmark of human linguistic systematicity.
Current Large Language Models (LLMs), while proficient at statistical pattern matching, often struggle with this type of systematicity. They tend to predict word meaning extensions based on literal similarity—chaining words that share physical features—rather than the underlying relational logic. This makes them brittle when encountering "radical" meaning shifts, such as those found in metaphors. We aim to bridge this gap by teaching models the "rules of the game" for how meanings jump between domains.
The SWORME Framework
To address this gap, we developed SWORME (Systematic Word Meta-Sense Extension), a framework designed to test and improve a model's ability to extend word meanings to new domains by recognizing patterns in other words.
The SWORME pipeline involves several key steps:
- Meta-sense Partitioning: Word senses are categorized into 39 high-level "meta-senses" (e.g.,
ITEM,COMMUNICATION,LOCATION) using the CoreLex ontology. - The "Sibling" Task: To simulate learning a new meaning, polysemous words are "split" into two hypothetical tokens (e.g.,
arrive_physicalandarrive_abstract). - Analogical Chaining: This is the core innovation. Instead of relying solely on associative similarity, the model is trained to recognize the relationship between domains using a "parallelogram" model of analogy: the vector offset between "physical grasp" and "mental grasp" should be similar to the offset between "physical steal" and "mental steal."
Key Insights and Results
Our research yielded several critical insights into how models handle semantic extension:
- Incremental vs. Radical Change: LMs are relatively effective at "incremental" changes where domains are conceptually close (e.g., metonymy like "chicken" as animal vs. meat). However, they struggle with "radical" changes like metaphors (e.g., "weathering a storm" vs. "weathering a crisis").
- Relational Logic Trumps Feature Similarity: Models perform significantly better when they learn the transformation between domains rather than just searching for nearest neighbors in a high-dimensional feature space.
- Generalization to Downstream Tasks: Fine-tuning on the SWORME task improved the performance of models like BERT, GPT-2, and LLaMA-7B on downstream figurative language benchmarks. For instance, LLaMA-7B's accuracy on the IMPLI dataset (metaphors and idioms) rose from 85.21% to 88.11% after SWORME training.
Conclusion and Future Directions
While SWORME significantly improves systematicity, challenges remain. Current models often treat meaning extension as bidirectional, whereas human language extension is frequently uni-directional (moving from concrete physical objects to abstract concepts). Future work could also explore incorporating historical context and more complex meaning changes like irony or sarcasm into the framework. By teaching models the systematic relationships between domains, we move closer to language technology that truly understands the creative flexibility of human expression.
Summary updated