TL;DR
This paper explores how well large language models (LLMs) can perform metaphor detection by directly translating the steps of the MIPVU manual procedure into prompts. While LLMs can mimic the procedure with minor compromises, they still fall short of supervised methods, particularly in distinguishing literal from metaphorical language. However, LLaMa shows promise in identifying similarities between literal and metaphorical meanings, suggesting potential for future research.
What Problem Does This Paper Address?
Metaphors are fundamental to human cognition and language, and their automatic detection is a crucial task in Natural Language Processing (NLP). Traditional approaches often tried to model the guidelines of manual metaphor identification. This paper investigates a newer approach: directly instructing LLMs to perform the steps of a well-established manual metaphor identification procedure, the Metaphor Identification Procedure VU Amsterdam (MIPVU). The core challenge is to see if LLMs, with their advanced reasoning capabilities, can effectively replicate the nuanced judgments required by MIPVU, especially in distinguishing between literal and metaphorical language and identifying the underlying analogical relationships.
Methods
The authors adapted the MIPVU procedure for use with LLMs by creating specific prompts for each step. This involved:
- Contextual Meaning Identification: Prompting the LLM to describe the meaning of a target word within its given context.
- Basic Meaning Identification: Asking the LLM to generate a dictionary entry for the word and then identify if any of its senses are more concrete than the contextual meaning. This step required compromises, as the LLM was asked to identify a concrete sense rather than exhaustively checking all possibilities.
- Distinctness and Similarity Judgment: Prompting the LLM to determine if the contextual and basic meanings are distinct and, if so, if they share similarities (aspects, functions, or features). They tested both zero-shot and one-shot prompting for similarity, with the latter providing examples.
To further improve performance, they also explored a hybrid approach combining a BERT-based word-sense disambiguation (WSD) model with LLMs.
Datasets
The authors evaluated their approach on two large-scale metaphor datasets:
- VUA dataset: Based on the original application of MIPVU to the British National Corpus.
- R&S dataset: Consisting of posts from Christian subreddits, annotated for metaphor using MIPVU.
Key Insights and Ideas
- LLMs can mimic MIPVU, but with limitations: The study demonstrates that LLMs can follow the steps of MIPVU through prompting, but they struggle to achieve the performance of supervised, fine-tuned models.
- Literal vs. Metaphorical Distinction is a major hurdle: A key weakness identified is the LLMs' difficulty in filtering out literal examples, leading to a high rate of false positives. This is attributed to failures in earlier MIPVU steps, which then propagate errors.
- LLaMa shows promise in analogical reasoning: Despite overall limitations, the LLaMa models exhibited interesting potential in identifying similarities between literal and metaphorical meanings, particularly when provided with examples. This suggests that LLMs might be better at the "similarity" aspect of metaphor detection than other steps.
- Prompting strategy matters: The performance of LLMs varied significantly based on the prompting strategy, with one-shot prompting (providing examples) generally improving recall for LLaMa.
Key Results
- Overall performance: None of the evaluated LLMs (LLaMa 3.1 8B and 70B, GPT-40-mini, Mistral-Small) matched the performance of supervised, fine-tuned approaches.
- False Positives: LLaMa and Mistral produced a significant number of false positives, indicating they struggled to correctly identify non-metaphorical instances.
- False Negatives: Heavily conventionalized metaphors were often missed (false negatives), particularly by Mistral when using WordNet glosses.
- Impact of WSD: Using a BERT-based WSD model improved precision but often led to a drop in recall.
- LLaMa's strengths: LLaMa models showed better recall when provided with examples for the similarity step.
Limitations and Future Work
- Problematic Steps: The authors highlight that identifying a "more basic" meaning and judging the distinctness of word senses remain significant challenges for LLMs.
- Vagueness in MIPVU: The definition of "more concrete" in MIPVU is noted as being somewhat vague, suggesting that providing more comprehensive definitions or concreteness ratings could improve LLM performance.
- Need for Gold Labels: The authors suggest that more detailed human judgments for all MIPVU steps (not just final labels) are needed for a better evaluation of LLMs.
- Future Directions: Future work should focus on improving LLMs' ability to discern concreteness and distinctness. Further investigation into analogical reasoning capabilities, including more systematic prompting and exploring different example sets, is also recommended. The authors also suggest that LLMs might be able to conduct MIPVU more effectively if they are explicitly prompted to identify source and target terms, similar to other recent work.
Summary updated