A Theory Guided Scaffolding Instruction Framework for LLM-Enabled Metaphor Reasoning

Yuan Tian, Nan Xu, Wenji MaoNAACL 2024 · OpenReview

Abstract

Metaphor detection is a challenging task in figurative language processing, which aims to distinguish between metaphorical and literal expressions in text. Existing methods tackle metaphor detection via training or fine-tuning discriminative models on labeled data. However, these approaches struggle to explain the underlying reasoning process behind the metaphorical/literal judgment. Recently, large language models (LLMs) have shown promise in language reasoning tasks. Although promising, LLM-based methods for metaphor detection and reasoning are still faced with the challenging issue of bringing the explainable concepts for metaphor reasoning and their linguistic manifestation. To fill this gap, we propose a novel Theory guided Scaffolding Instruction (TSI) framework that instructs an LLM to infer the underlying reasoning process of metaphor detection guided by metaphor theories for the first time. Our work is inspired by a pedagogical strategy called scaffolding instruction, which encourages educators to provide questioning and support as scaffolding so as to assist learners in constructing the understanding of pedagogical goals step by step. We first construct a metaphor knowledge graph grounded in metaphor theory which serves as the instructional structure to obtain a series of scaffolding questions, directing the LLM to incrementally generate the reasoning process for metaphor understanding through dialogue interactions. During this theory guided instruction process, we explore the LLM’s mastery boundary and provide the relevant knowledge as scaffolding support when the question is beyond the LLM’s capability. Experimental results verify that our method significantly outperforms both the LLM-based reasoning methods and the SOTA methods in metaphor detection, indicating the facilitation of metaphor and instruction theories in guiding LLM-based reasoning process.

Introduction

Metaphor detection—the ability to distinguish between literal expressions (e.g., "wearing a shirt") and metaphorical ones (e.g., "wearing a smile")—is a cornerstone of natural language understanding. While traditional AI models can be trained to spot these patterns, they often function as "black boxes," offering no explanation for the underlying reasoning. Large Language Models (LLMs) like GPT-3.5 show promise in reasoning, but they frequently stumble on metaphors because these expressions represent cognitive mappings between different domains of experience rather than simple linguistic quirks. Standard prompting techniques, such as "let's think step by step," often fail because the model lacks the theoretical framework necessary to navigate the complex relationship between a "source domain" and a "target domain."

The TSI Framework

To address these limitations, we introduce the Theory guided Scaffolding Instruction (TSI) framework. Inspired by the pedagogical concept of "scaffolding," where an educator provides temporary support to help a learner reach a goal, TSI guides LLMs through the reasoning process using structured linguistic theories.

The Theory guided Scaffolding Instruction (TSI) framework architecture.

The framework is built upon four main pillars:

  1. Metaphor Knowledge Graph (KG) Construction: We simplified three major metaphor theories—Conceptual Metaphor Theory (CMT), Metaphor Identification Procedure (MIP), and Selectional Preference Violation (SPV)—into structured Knowledge Graphs that define entities (words, domains, meanings) and their relationships.
  2. Scaffolding Question Construction: Using the KG as a map, the system generates a sequence of questions. For instance, it might ask the LLM to identify the source domain of a word, then the target domain, and finally determine if they differ.
  3. Mastery Level Support: The system dynamically checks the LLM's "mastery boundary" by asking questions multiple times. If answers are inconsistent (measured by a Consistency Ratio), the system provides a "scaffold"—a snippet of theoretical knowledge from the KG—to assist the model.
  4. Classification: The final judgment is derived from whether the LLM's reasoning process aligns with the structure of the chosen metaphor theory.

Experimental Evaluation

We evaluated TSI on two widely used metaphor detection benchmarks: MOH-X (647 single-verb instances) and TroFi (3,737 verb-based instances from the Wall Street Journal). For our experiments, we utilized balanced subsets of 300 instances from each dataset.

Performance comparison of TSI variants and ablation studies on MOH-X and TroFi datasets.

As shown in the results table, TSI (using CMT) achieved significant performance gains. On MOH-X, TSI (CMT) reached an F1 score of 82.59, outperforming the best LLM baseline (Least-to-most at 74.43) and specialized models like AdMul (79.74). On TroFi, TSI (CMT) achieved an F1 of 66.07, surpassing Zero-shot CoT (64.18) and SOTA discriminative models like MelBERT (62.36). Ablation studies confirmed that removing the "Mastery Level Verifier" or the "Knowledge Scaffolding" led to a drop in performance, proving the necessity of the dynamic support system.

Key Insights

  • Theory as a Cognitive Shortcut: Providing a theoretical framework like CMT helps the model map abstract concepts to concrete ones more effectively than generic logical prompts.
  • Structured Knowledge over Text: LLMs appear to internalize metaphor theories better through structured Knowledge Graphs than through long-form textual descriptions.
  • CMT Superiority: Among the theories tested, Conceptual Metaphor Theory consistently performed best, suggesting that domain mapping is the most effective approach for AI metaphor processing.
  • Consistency as Mastery: Using a Consistency Ratio effectively identifies when a model is guessing, allowing for targeted scaffolding support.

Conclusion and Future Work

While TSI significantly advances metaphor reasoning, it faces challenges regarding computational cost due to multiple LLM calls and the manual effort required to simplify linguistic theories into KGs. Future work will explore automating KG construction and scaling the framework to larger, more diverse datasets to further refine the model's figurative language understanding.

Summary updated

Abstract

Metaphor detection is a challenging task in figurative language processing, which aims to distinguish between metaphorical and literal expressions in text. Existing methods tackle metaphor detection via training or fine-tuning discriminative models on labeled data. However, these approaches struggle to explain the underlying reasoning process behind the metaphorical/literal judgment. Recently, large language models (LLMs) have shown promise in language reasoning tasks. Although promising, LLM-based methods for metaphor detection and reasoning are still faced with the challenging issue of bringing the explainable concepts for metaphor reasoning and their linguistic manifestation. To fill this gap, we propose a novel Theory guided Scaffolding Instruction (TSI) framework that instructs an LLM to infer the underlying reasoning process of metaphor detection guided by metaphor theories for the first time. Our work is inspired by a pedagogical strategy called scaffolding instruction, which encourages educators to provide questioning and support as scaffolding so as to assist learners in constructing the understanding of pedagogical goals step by step. We first construct a metaphor knowledge graph grounded in metaphor theory which serves as the instructional structure to obtain a series of scaffolding questions, directing the LLM to incrementally generate the reasoning process for metaphor understanding through dialogue interactions. During this theory guided instruction process, we explore the LLM’s mastery boundary and provide the relevant knowledge as scaffolding support when the question is beyond the LLM’s capability. Experimental results verify that our method significantly outperforms both the LLM-based reasoning methods and the SOTA methods in metaphor detection, indicating the facilitation of metaphor and instruction theories in guiding LLM-based reasoning process.
BibTeX
@inproceedings{tian2024theory,
  author = {Yuan Tian and Nan Xu and Wenji Mao},
  title = {{A Theory Guided Scaffolding Instruction Framework for LLM-Enabled Metaphor Reasoning}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=0rleTiw1s6},
}
Figure