UCSC NLP T6 at SemEval-2025 Task 1: Leveraging LLMs and VLMs for Idiomatic Understanding

Judith Clymo, Adam Zernik, Shubham GaurACL 2025

TL;DR

This paper tackles the challenge of teaching AI to understand idiomatic language, which often defies literal interpretation. The researchers leverage Large Language Models (LLMs) and Vision-Language Models (VLMs) to not only identify idiomatic phrases but also to match them with relevant images, demonstrating a promising step towards more nuanced AI comprehension.

What Problem Does This Paper Address?

Idiomatic expressions, like "a piece of cake" (meaning something easy, not actual dessert), are a significant hurdle for natural language processing (NLP) models. Their meanings are non-compositional, meaning they can't be deduced from the individual words. This makes it difficult for AI to grasp the intended meaning, impacting applications like machine translation, sentiment analysis, and dialogue systems. The SemEval-2025 Task 1 (AdMIRe) specifically challenges models to distinguish between literal and idiomatic uses of phrases and to find images that best represent the idiomatic meaning. This task is crucial for developing AI that can understand the richness and subtlety of human language.

Methods

The UCSC NLP T6 team employed a multi-modal approach, combining the strengths of LLMs and VLMs.

  1. Sentence Type Classification: They first used GPT-4 to classify whether a target phrase in a given sentence was used literally or idiomatically. This initial step is crucial for tailoring subsequent processing.
  2. Semantic Enhancement of Text Inputs: For idiomatic phrases, they used GPT-4 with carefully crafted prompts to generate context-aware definitions. These prompts employed techniques like manual template engineering, prompt augmentation with few-shot examples, chain-of-thought reasoning, and prompt composition to elicit more accurate and descriptive definitions. The goal was to create text inputs for the VLMs that captured the figurative meaning of the idiom.
  3. Image Alignment: The generated definitions (or the original phrases for literal uses) were then fed into VLMs. The researchers experimented with several VLMs, including CLIP, ALIGN, and OpenCLIP, which are trained to match text and image embeddings. These models generate embeddings for both text and images, and their similarity is measured using cosine similarity. The system then ranks the provided images based on how well their embeddings align with the text input's embedding.

Datasets

The research utilized data from the SemEval-2025 Task 1 (AdMIRe) dataset, which includes context sentences with target phrases, five associated images, and ground truth labels for literal/idiomatic usage and correct image rankings. The task was available in both English and Portuguese, but this work focused solely on the English version.

Key Insights and Ideas

  • GPT-4 as a Versatile Tool: GPT-4 proved highly effective not only for generating nuanced definitions of idioms but also for the initial classification of literal vs. idiomatic usage, simplifying the overall pipeline.
  • The Challenge of Literal vs. Idiomatic Visuals: VLMs, often trained on image captioning tasks, exhibit a strong bias towards literal interpretations. This makes it challenging for them to accurately rank images for idiomatic phrases, as they tend to favor more direct, visual representations.
  • Prompt Engineering Matters: The choice of prompts significantly impacts the quality of definitions generated by LLMs. Prompts that encourage more specific, scene-based descriptions (like Prompt 2) can be more effective for image matching than purely abstract definitions.
  • Idiomaticity is Inherently Harder for VLMs: Even with sophisticated LLM-generated definitions, VLMs struggled more with idiomatic contexts compared to literal ones. This suggests that current VLMs may not fully grasp the abstract and cultural nuances inherent in idiomatic language.

Key Results

  • The team achieved strong performance in classifying sentence types using GPT-4.
  • For the Top-1 Accuracy metric on the test dataset, their best model achieved an accuracy of 0.87 for idiomatic phrases and 0.88 for literal phrases.
  • The results showed that while VLMs performed well on literal phrases, they struggled with idiomatic ones, often preferring literal interpretations even when the text clearly indicated an idiom.
  • Among the VLMs tested, ALIGN and OpenCLIP showed greater consistency across different metrics and datasets compared to CLIP variants.

Limitations and Future Work

  • LLM Inconsistency: A significant limitation is the inherent variability in LLM responses, even with identical prompts. This makes it difficult to fully understand the prompt-output relationship and can affect downstream tasks.
  • VLM Bias: The strong bias of VLMs towards literal interpretations remains a key challenge. Future work could explore methods to mitigate this bias, perhaps through specialized training data or more advanced prompt engineering for VLMs.
  • Model Updates: LLM updates can unpredictably alter performance, requiring continuous monitoring and prompt adjustments.
  • Future Directions: The authors suggest exploring more complex voting algorithms for combining multiple LLM attempts and developing automated prompt update mechanisms. They also highlight the broad applicability of their techniques, such as using LLMs for input rephrasing and chaining LLM calls in agentic frameworks.

Summary updated

BibTeX
@inproceedings{clymo2025ucsc,
  author = {Judith Clymo and Adam Zernik and Shubham Gaur},
  title = {{UCSC NLP T6 at SemEval-2025 Task 1: Leveraging LLMs and VLMs for Idiomatic Understanding}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure