UMUTeam at SemEval-2025 Task 1: Leveraging Multimodal and Large Language Model for Identifying and Ranking Idiomatic Expressions

Ronghao Pan, Tomás Bernal-Beltrán, José Antonio García-Díaz, Rafael Valencia-GarcíaACL 2025

TL;DR

This paper tackles the challenge of understanding idiomatic expressions, which often defy literal interpretation. The UMUTeam developed multimodal and large language model (LLM) approaches to rank images based on how well they represent the meaning of an idiom in context, achieving competitive results in the SemEval-2025 AdMIRe task.

What Problem Does This Paper Address?

Idioms are phrases whose meaning cannot be directly inferred from the individual words. This makes them a significant hurdle for natural language processing (NLP) systems, which tend to process language literally. While LLMs have shown promise in various NLP tasks, they often struggle with idioms, leading to misinterpretations and translation errors. The AdMIRe shared task (SemEval 2025) specifically addresses this by focusing on understanding idioms through visual and contextual representations, moving beyond simple binary classification to a more nuanced task of associating meanings with images. This is crucial for applications like sentiment analysis and machine translation where accurate comprehension of figurative language is essential.

Methods

The UMUTeam explored two primary approaches for Subtask A of the AdMIRe task, which involves ranking images based on their semantic alignment with an idiomatic expression in a given context sentence.

  1. Multimodal Approach (BERT/XLM-ROBERTa + ViT): This method combines textual and visual features.
  2. Textual Encoding: Pre-trained language models like BERT (for English) and XLM-ROBERTa (for English and Portuguese) were used to generate contextual embeddings for the sentence and image captions.
  3. Visual Encoding: Vision Transformers (ViT) were employed to extract features from images.
  4. Fusion: A gated fusion mechanism based on multi-head attention was used to dynamically combine the textual and visual embeddings, creating a unified multimodal representation.
  5. Ranking: This fused representation was then fed into a regression layer to predict a relevance score for each image, indicating how well it represents the idiom's meaning. The model was trained to minimize Root Mean Square Error (RMSE).
  6. LLM In-Context Learning Approach (Llama-3.1-8B-Instruct): This approach leverages the zero-shot learning capabilities of LLMs.
  7. Prompt Engineering: A prompt was constructed containing the context sentence and the captions for the five candidate images.
  8. Image Classification: The Llama-3.1-8B model was used to evaluate each image based on its description and generate a similarity score relative to the idiom's contextual meaning.
  9. Ranking: Images were then sorted based on these scores.

Datasets

The study utilized the training and test datasets provided for Subtask A of the SemEval-2025 AdMIRe shared task. These datasets consist of context sentences, nominal compounds (potential idioms), and five associated images with captions for each instance. The task involved ranking these images based on their relevance to the idiom's meaning in context.

Key Insights and Ideas

  • Multimodality is Key: The success of both approaches highlights the critical role of integrating visual information with textual context for understanding idioms. Idioms often rely on metaphorical or symbolic imagery that can be captured by visual representations.
  • Language-Specific Nuances Matter: The performance differences between English and Portuguese suggest that language-specific models and strategies are important for idiom interpretation. What works best in one language may not be optimal for another.
  • LLMs as Powerful Zero-Shot Learners: The Llama-3.1-8B model demonstrated strong performance, particularly in English, showcasing the potential of LLMs for idiom understanding through in-context learning without extensive task-specific fine-tuning.
  • The Challenge of Literal vs. Figurative: The core difficulty lies in training models to distinguish and correctly interpret the figurative meaning of idioms, rather than defaulting to their literal interpretations.

Key Results

The UMUTeam participated in Subtask A of the AdMIRe shared task, evaluating their models on both a test set and an extended evaluation set, in English and Portuguese.

  • English:
    • The Llama-3.1-8B-Instruct model achieved the best results for English, ranking 17th on the test set with an accuracy of 0.4 and a DCG score of 2.677, and 12th on the extended evaluation set with an accuracy of 0.24 and a DCG score of 2.520.
  • The BERT + ViT model achieved an accuracy of 0.266 and a DCG score of 2.337 on the test set.
  • The XLM-ROBERTa + ViT model achieved an accuracy of 0.133 and a DCG score of 2.232 on the test set.
  • Portuguese:
    • The XLM-ROBERTa + ViT model performed best for Portuguese, ranking 9th on the test set with an accuracy of 0.384 and a DCG score of 2.572, and 8th on the extended evaluation set with an accuracy of 0.182 and a DCG score of 2.331.
  • The BERT + ViT model achieved an accuracy of 0.308 and a DCG score of 2.475 on the test set.
  • The Llama-3.1-8B-Instruct model achieved an accuracy of 0.231 and a DCG score of 2.444 on the test set.

Limitations and Future Work

  • Limitations:
    • The official evaluation system prioritized English performance, which may have impacted the UMUTeam's overall ranking in Portuguese when using the Llama model.
  • While the multimodal approach is effective, the paper doesn't delve deeply into the specific types of visual cues that are most informative for idiom interpretation.
  • Future Work:
    • Explore more advanced fusion techniques for multimodal representations.
  • Investigate the role of cultural context in idiom interpretation to further improve model performance across languages.
  • The authors suggest that if the XLM + ViT model had been the primary approach for Portuguese, their ranking would have been significantly higher, indicating potential for further optimization by tailoring models to specific languages.

Summary updated

BibTeX
@inproceedings{pan2025umuteam,
  author = {Ronghao Pan and Tomás Bernal-Beltrán and José Antonio García-Díaz and Rafael Valencia-García},
  title = {{UMUTeam at SemEval-2025 Task 1: Leveraging Multimodal and Large Language Model for Identifying and Ranking Idiomatic Expressions}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure