Zhoumou at SemEval-2025 Task 1: Leveraging Multimodal Data Augmentation and Large Language Models for Enhanced Idiom Understanding

Yingzhou Zhao, Bowen Guan, Liang Yang, Hongfei LinACL 2025

TL;DR

This paper tackles the challenge of understanding idioms in images by combining multimodal data augmentation with advanced large language models (LLMs). They achieved strong results by using BLIP-2 to generate descriptive captions for images and then feeding these, along with original text, into the powerful DeepSeek LLM for relevance ranking.

What Problem Does This Paper Address?

Understanding figurative language, especially idioms, is a known weakness for current AI models. This is particularly true when idioms are presented visually, as humans often rely on multisensory interactions and real-world understanding to grasp their meaning. SemEval-2025 Task 1 specifically aims to improve how AI ranks images based on their relevance to a given noun compound (which can be an idiom) within a sentence. The core challenge is bridging the gap between visual content and the nuanced, often non-literal, meanings of idiomatic expressions.

Methods

The authors propose a two-stage system:

  1. Data Augmentation: They leverage the BLIP-2 multimodal model to generate descriptive text for candidate images. To optimize this, they fine-tune BLIP-2 using LoRA (Low-Rank Adaptation) on the provided training data. Interestingly, they found that fine-tuning on a smaller subset of 100 image-text pairs yielded more creative and detailed textual descriptions than fine-tuning on the full dataset. These generated captions are then integrated with the original image descriptions.
  2. Relevance Assessment: This stage transforms the multimodal problem into a text-based one.
  3. Caption Integration: A naive concatenation of original and generated captions led to redundancy. To overcome this, they use the DeepSeek LLM to synthesize these texts into a coherent, concise, and richly detailed representation, prompting it to act as a "master of textual integration and reasoning."
    • Direct API Calls: The integrated text, along with the target noun compound and its sentence context, is fed into the DeepSeek API. DeepSeek is prompted to act as an "image relevance ranking expert," considering literal and figurative meanings, and providing a rationale for its ranking.
  4. Fine-tuning LLMs (Exploratory): As an alternative, they also explored fine-tuning smaller LLMs (Llama-3.1-8B and Qwen2.5-7B) on the augmented textual data.

Datasets

The experiments were conducted on the SemEval-2025 Task 1 Subtask A dataset, which includes:

  • English Dataset: 70 data instances.
  • Portuguese Dataset: 32 data instances.

Each data instance consists of a compound word, a sentence using the word, the meaning type of the compound, five comic pictures, and a text description for each picture.

Key Insights and Ideas

  • Fine-tuning BLIP-2 for Better Descriptions: Fine-tuning BLIP-2, especially on a smaller, curated subset, can lead to more creative and detailed image descriptions, which are crucial for downstream tasks.
  • LLM for Text Synthesis is Key: Simply combining text sources isn't enough. Using a powerful LLM like DeepSeek to synthesize original and generated descriptions creates a more robust and semantically rich representation.
  • DeepSeek's Reasoning Power is Superior: For this specific task, directly using a large, externally hosted LLM like DeepSeek (with its inherent reasoning capabilities) significantly outperformed fine-tuning smaller models. This highlights the advantage of broad knowledge and sophisticated inference for nuanced tasks.
  • Data Augmentation Enhances LLM Performance: The multimodal data augmentation (image-to-text generation) significantly improved the LLM's ability to perform relevance assessments, even when the LLM itself wasn't multimodal.

Key Results

The system achieved strong ranking accuracy:

  • English Dataset: 0.73 accuracy.
  • Portuguese Dataset: 0.85 accuracy.

These results substantially outperformed the fine-tuned smaller LLMs (Llama-3.1-8B and Qwen2.5-7B), which achieved accuracies of 0.37 and 0.34 (English) and 0.27 and 0.29 (Portuguese), respectively. The authors note that the DeepSeek API, when provided with the textually augmented data, generated more accurate relevance rankings than using original image-text data alone.

Limitations and Future Work

  • Limitations of Fine-tuning: The paper acknowledges that fine-tuning smaller models, while potentially more resource-efficient, struggled to match the performance of large, externally hosted LLMs for this complex task requiring nuanced semantic understanding and reasoning. They found that these smaller models had difficulty generalizing to the complexities of idiomatic expressions and subtle contextual cues.
  • Future Work: The authors suggest future research should focus on:
  • Exploring further possibilities for multimodal idiom understanding.
  • Enhancing the direct comprehension capabilities of multimodal LLMs for both image and text.
  • Investigating text-to-image generation techniques to facilitate more effective text-based ranking.
  • Exploring alternative fine-tuning techniques for smaller models, such as more extensive fine-tuning, incorporating diverse training data, or using specialized loss functions.

Summary updated

BibTeX
@inproceedings{zhao2025zhoumou,
  author = {Yingzhou Zhao and Bowen Guan and Liang Yang and Hongfei Lin},
  title = {{Zhoumou at SemEval-2025 Task 1: Leveraging Multimodal Data Augmentation and Large Language Models for Enhanced Idiom Understanding}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure