YNU-HPCC at SemEval-2025 Task 1: Enhancing Multimodal Idiomaticity Representation via LoRA and Hybrid Loss Optimization

Liu Lei, You Zhang, Jin Wang, Dan Xu, Xuejie ZhangACL 2025

TL;DR

This paper tackles the challenge of multimodal idiomaticity representation, where models must match idioms in sentences to relevant images, a task complicated by literal vs. figurative meanings. The YNU-HPCC team improved upon the AltCLIP-m18 model by using Low-Rank Adaptation (LoRA) for efficient fine-tuning and a novel hybrid loss function combining ListMLE and Focal Loss, achieving state-of-the-art results on the SemEval-2025 Task 1.

What Problem Does This Paper Address?

Understanding idioms is a notoriously difficult problem in natural language processing. Idioms, or multiword expressions (MWEs), have meanings that cannot be deduced from the individual words they comprise. For example, "kick the bucket" doesn't literally involve kicking anything, but rather means to die. This inherent ambiguity makes it challenging for AI models to grasp their true meaning. The SemEval-2025 Task 1, specifically Subtask A, pushes this challenge further by requiring models to not only understand the idiomatic meaning from text but also to align it with visual representations. This involves disambiguating between literal and figurative interpretations of an idiom within a sentence and then selecting the most semantically relevant image from a set of candidates. This is particularly tricky because models might default to literal interpretations or struggle to connect abstract idiomatic concepts to concrete visual cues, especially across different languages like English and Portuguese.

Methods

The YNU-HPCC team built their solution upon the AltCLIP-m18 model, a multilingual extension of the CLIP (Contrastive Language-Image Pre-training) model. AltCLIP-m18 is capable of mapping text and images into a shared embedding space, making it a strong foundation for multimodal tasks.

To make the fine-tuning process more efficient, they employed Low-Rank Adaptation (LoRA). Instead of retraining all parameters of the large AltCLIP-m18 model, LoRA injects small, trainable low-rank matrices into specific layers (query and value projection matrices in the Transformer layers). This significantly reduces the number of trainable parameters, leading to faster training and lower computational costs while aiming to preserve performance.

A key innovation is their hybrid loss function. They combined two loss functions:

  • ListMLE Loss: This is a listwise learning-to-rank (LTR) method that optimizes the global ranking of candidate images. It aims to ensure that the correct image is ranked highest among all options.
  • Focal Loss: This loss function is designed to address hard-to-classify samples, which are crucial in this task where distinguishing between literal and idiomatic meanings can be difficult. Focal Loss dynamically down-weights easy examples and focuses training on harder ones, helping the model better differentiate between subtle semantic nuances.

The final loss is a weighted combination of these two, controlled by a balancing factor $\lambda$. Additionally, they used a data augmentation strategy involving generating sentence variants with inverted sentence types (idiomatic vs. literal) to improve the model's robustness and ability to handle ambiguity.

Datasets

The primary dataset used for this task is from SemEval-2025 Task 1 Subtask A. This dataset includes:

  • 70 English training items and 32 Portuguese training items.
  • Each item consists of a context sentence with a potentially idiomatic nominal compound (NC), and five candidate images.
  • The images are categorized into five types: synonym for idiomatic meaning, synonym for literal meaning, related to idiomatic meaning (but not synonymous), related to literal meaning (but not synonymous), and a distractor.
  • For training, sentence types and ground-truth image order are provided. For development and test sets, these are withheld, requiring the model to predict the correct order.

Key Insights and Ideas

  • Efficiency is Key: LoRA proves to be a highly effective method for adapting large multimodal models like AltCLIP-m18 to specific tasks without the prohibitive computational cost of full fine-tuning. This opens doors for more accessible research and application of powerful models.
  • Hybrid Loss for Nuance: The combination of ListMLE and Focal Loss is crucial. ListMLE handles the overall ranking task, while Focal Loss specifically targets the difficult cases of distinguishing literal from idiomatic meanings, which is the core challenge of this task.
  • Data Augmentation Matters: Generating sentence variants that flip the intended meaning (idiomatic to literal, or vice-versa) is a clever way to force the model to learn finer distinctions and improve its generalization.

Key Results

The YNU-HPCC team achieved significant improvements over their baseline AltCLIP-m18 model. On the primary test set:

  • English: They achieved a Top-1 Accuracy of 0.53 and a DCG of 2.94, outperforming the baseline's 0.40 Top-1 Accuracy and 2.98 DCG.
  • Portuguese: They achieved a Top-1 Accuracy of 0.77 and a DCG of 3.31, surpassing the baseline's 0.55 Top-1 Accuracy and 2.98 DCG.

These results demonstrate the effectiveness of their LoRA fine-tuning and hybrid loss optimization strategy.

Limitations and Future Work

The authors acknowledge a key limitation: the relatively small size of the training dataset, particularly for Portuguese. They suggest that this might affect the generalizability of their results. Future work could involve expanding the dataset or exploring transfer learning techniques to leverage larger, related datasets to further improve performance.

Summary updated

BibTeX
@inproceedings{lei2025ynu,
  author = {Liu Lei and You Zhang and Jin Wang and Dan Xu and Xuejie Zhang},
  title = {{YNU-HPCC at SemEval-2025 Task 1: Enhancing Multimodal Idiomaticity Representation via LoRA and Hybrid Loss Optimization}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure