Easy as PIE? Identifying Multi-Word Expressions with LLMs

Kai Golan Hashiloni, Ofri Hefetz, Kfir BarEMNLP 2025

TL;DR

This paper explores using large language models (LLMs) for identifying idiomatic expressions in text without any fine-tuning, relying solely on clever prompting. They demonstrate that well-crafted prompts allow LLMs to perform competitively with traditional supervised models, highlighting their potential as a flexible tool for this challenging NLP task.

What Problem Does This Paper Address?

Multi-word expressions (MWEs), especially idioms, are a persistent challenge in Natural Language Processing (NLP). These are phrases where the meaning of the whole isn't simply the sum of its parts (e.g., "kick the bucket"). Misinterpreting idioms can significantly harm downstream tasks like machine translation or semantic analysis. While LLMs have revolutionized many NLP areas, their ability to identify idioms directly from text, without task-specific training, remains underexplored. This paper aims to bridge that gap by investigating how generative LLMs can be prompted to detect these non-compositional expressions.

Methods

The core of this research lies in prompt engineering. Instead of fine-tuning LLMs on specific idiom datasets, the authors employ a prompt-based approach. They experiment with various prompting strategies, including:

  • Zero-shot: The LLM is given the task instructions and the sentence, with no examples.
  • Few-shot: The LLM is provided with a small number of example idiom-sentence pairs to guide its understanding.
  • Chain-of-Thought (CoT): This technique encourages the LLM to "think step-by-step" by generating intermediate reasoning or explanations before providing the final answer. They explore variations like CoTGen (generating examples) and CoTBest (selecting the most confident idiom).
  • Self-Consistency (SC): The model is run multiple times with the same prompt, and the consensus output is taken to improve stability.

They also explore MWE identification beyond just idioms, using a broader dataset.

Datasets

The study utilizes several datasets for evaluation:

  • ID10M: A multilingual dataset for idiom identification, featuring automatically generated training data and a manually annotated evaluation benchmark in English, Italian, and German.
  • MAGPIE: An English dataset originally designed for idiom classification, adapted for identification by concatenating context sentences.
  • CoAM: A dataset designed for comprehensive MWE identification, covering various MWE categories beyond just idioms.

Key Insights and Ideas

  • LLMs are surprisingly capable with just prompts: The most striking finding is that LLMs, when prompted effectively, can achieve performance comparable to or even better than supervised models trained on annotated data, even without fine-tuning.
  • Prompting is key: The choice of prompt significantly impacts performance. CoT reasoning, in particular, seems to boost performance by guiding the LLM's understanding.
  • Idioms are tricky, even for LLMs: While LLMs show promise, they still struggle with nuances like boundary errors (misidentifying the exact span of an idiom) and literal-as-idiomatic false positives, especially in morphologically rich languages.
  • MWE identification is harder than idiom identification: Moving from specific idioms to the broader category of MWEs leads to a performance drop, likely due to the more flexible and ambiguous definitions of MWEs and potential annotation biases.
  • Reasoning models show promise: Specialized reasoning models, like DeepSeek-R1, perform well even in zero-shot settings, suggesting that built-in reasoning capabilities are valuable.

Key Results

  • On the ID10M dataset, at least one LLM configuration outperformed the baseline in every language, often using smaller, cost-efficient models. Few-shot prompting consistently improved performance, and CoT reasoning further enhanced it.
  • For MAGPIE, CoT reasoning generally improved performance, though complex variants sometimes reduced it. Reasoning-focused models achieved the best results.
  • On the CoAM dataset (broader MWE identification), LLMs performed below fine-tuned state-of-the-art models, but their zero-shot configurations remained competitive. Few-shot prompting surprisingly decreased performance here, while self-consistency was more helpful.
  • Qualitative analysis revealed common error types: literal-as-idiomatic, missed predictions, boundary errors, and modification errors (where the LLM alters the idiom's surface form).

Limitations and Future Work

The authors acknowledge several limitations:

  • The ID10M dataset has a small number of examples and only continuous idioms.
  • Due to budget constraints, they used a limited subset of configurations and models.
  • Reproducibility depends on proprietary model providers.
  • There's a potential concern that LLMs might have been exposed to the training data during their pre-training.
  • The MAGPIE dataset was evaluated on a subset, and no direct baseline was available for comparison.
  • Manual prompt optimization might not be optimal for every individual model.

Future work could involve broader evaluations across more datasets, idiom types, and languages, as well as exploring automatic prompt optimization and model-specific tuning. Investigating reasoning models further for MWE identification is also a promising direction.

Summary updated

BibTeX
@inproceedings{hashiloni2025easy,
  author = {Kai Golan Hashiloni and Ofri Hefetz and Kfir Bar},
  title = {{Easy as PIE? Identifying Multi-Word Expressions with LLMs}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure