TL;DR
This paper tackles the challenge of representing idiomatic expressions in a multimodal context (text and images) by leveraging generative Large Language Models (LLMs) and CLIP models. They propose a method that uses LLMs to generate richer, idiomatic meanings for phrases, which are then encoded by CLIP for improved image ranking, outperforming baseline approaches.
What Problem Does This Paper Address?
Understanding and representing idiomatic expressions is a long-standing challenge in Natural Language Processing (NLP). Idioms, by their nature, have meanings that cannot be deduced from the literal meanings of their constituent words. This non-compositional aspect makes it difficult for standard NLP models, which often rely on surface-level word associations, to grasp their true meaning. SemEval-2025 Task 1 specifically focuses on this problem in a multimodal setting, requiring systems to rank images based on how well they represent a potentially idiomatic nominal compound within a given context sentence, in both English and Brazilian Portuguese. The core challenge is to bridge the gap between the abstract, non-literal meaning of an idiom and its visual representation.
Methods
The proposed method combines generative LLMs with multilingual CLIP models in a multi-stage process:
- Idiomatic Meaning Generation: An LLM is first used to classify whether a given compound phrase is used literally or idiomatically within its context. If classified as idiomatic, the LLM is then prompted to generate a descriptive meaning of the idiom. This step aims to create a more explicit and semantically rich representation of the idiom's meaning. To improve robustness, this classification is repeated multiple times, and a majority vote is taken.
- Embedding Extraction and Image Ranking: A multilingual CLIP model is employed to extract embeddings. Crucially, if a compound is deemed idiomatic, its embedding is derived from the generated idiomatic meaning, not the original phrase. For literal compounds, the original phrase's embedding is used. Image embeddings are extracted for candidate images, and caption embeddings are also generated (with truncation to fit model input limits). A ranking score is then computed by summing the similarity between the compound embedding and both the image embedding and the caption embedding.
- Ensemble Method: To further enhance robustness, the authors propose an ensemble approach where multiple LLMs generate idiomatic meanings, and their individual ranking scores are averaged to produce a final score. This aims to smooth out individual model biases.
- Fine-Tuning with Contrastive Learning: To improve the alignment between idiomatic compound representations and their corresponding images, the extracted CLIP embeddings are fine-tuned using a contrastive learning approach. This involves creating anchor-positive-negative triplets and training a model to maximize similarity between positive pairs (e.g., compound and its correct image) and minimize it for negative pairs. Data augmentation techniques (image transformations, back-translation, paraphrasing) are applied to the training data to improve the model's robustness.
Datasets
The experiments were conducted on two datasets provided for SemEval-2025 Task 1, Subtask A:
- English Dataset: 70 training, 15 development, 15 test, and 100 extended test samples.
- Brazilian Portuguese Dataset: 32 training, 10 development, 13 test, and 55 extended test samples.
For fine-tuning, augmented training sets were used, based on ground-truth compound types from the original training sets to avoid misclassification errors during this phase.
Key Insights and Ideas
- LLMs as Idiom Interpreters: The core idea of using LLMs to explicitly generate idiomatic meanings is a powerful way to overcome the limitations of surface-level understanding in standard models. This provides a richer semantic signal for downstream tasks.
- Multimodal Alignment via Generated Meaning: Deriving CLIP embeddings from the generated idiomatic meaning rather than the literal phrase is key to improving multimodal alignment. This allows the model to connect abstract concepts to visual representations more effectively.
- Fine-tuning Challenges: While fine-tuning with contrastive learning is a standard technique for improving representation alignment, the paper found that it did not outperform non-fine-tuned embeddings in this specific task. This suggests potential issues with data size, augmentation strategies, or the inherent complexity of fine-tuning for idiomaticity.
- Language and Model Specificity: The performance of different LLMs (GPT-3.5, GPT-4, GPT-40) and CLIP models (XLM-14, XLM-32, LABSE-14) varied across languages and tasks, highlighting the need for careful selection and ensemble approaches. GPT-4 generally showed superior performance in compound type detection.
Key Results
- The proposed method, using generative LLMs and CLIP models, outperformed baseline approaches that relied solely on original nominal compounds for image ranking.
- For English, an ensemble method combining GPT-3.5, GPT-4, and GPT-40 with the XLM-R Large ViT-B/32 CLIP model achieved the best results.
- For Brazilian Portuguese, GPT-3.5 with the LABSE ViT-L/14 CLIP model performed best.
- Fine-tuning CLIP embeddings (CI-F and CIC-F) resulted in worse performance compared to using non-fine-tuned embeddings. This was observed across various LLM and CLIP combinations.
- Compound type detection accuracy was highest with GPT-4 on both English and Portuguese datasets.
Limitations and Future Work
- Fine-tuning Performance: The primary limitation identified is that fine-tuning the CLIP embeddings did not improve performance and, in fact, led to worse results. The authors attribute this to potential issues with insufficient fine-tuning data, the model's capacity, or overfitting.
- Caption Utilization: The inclusion of caption embeddings in the CIC approach did not significantly improve results compared to CI. The authors suggest that the lengthy captions might have made their embeddings less effective.
- Future Work Suggestions:
- Improving caption utilization through different truncation methods or paraphrasing.
- Refining fine-tuning strategies and expanding the training data.
- Exploring more advanced contrastive learning techniques.
- Investigating the impact of different LLM architectures and CLIP model variants.
Summary updated