Ukrainian Visual Word Sense Disambiguation Benchmark

Yurii Laba, Yaryna Mohytych, Ivanna Rohulia, Halyna Kyryleyza, Hanna Dydyk-Meush, Oles Dobosevych, Rostyslav HrynivCOLING 2024

TL;DR

This paper introduces the first benchmark for Visual Word Sense Disambiguation (Visual-WSD) in Ukrainian, addressing the underperformance of multilingual models on low-resource languages. They found that current state-of-the-art models struggle significantly with Ukrainian Visual-WSD compared to English, highlighting a critical gap in multimodal AI capabilities.

What Problem Does This Paper Address?

Large Language Models (LLMs) and their multimodal counterparts (MLLMs) have made impressive strides in understanding and generating text and images. However, a significant challenge arises with "hallucinations," where models produce outputs that deviate from reality or user input. This is particularly problematic for low-resource languages like Ukrainian, where models often exhibit inferior performance. A specific instance of this is Visual Word Sense Disambiguation (Visual-WSD), where a model must identify the correct image corresponding to an ambiguous word in a given context. Existing Visual-WSD benchmarks are primarily available for high-resource languages like English, Italian, and Farsi, leaving a gap in evaluating and improving MLLMs for Ukrainian. This paper aims to bridge this gap by creating a Ukrainian Visual-WSD benchmark and evaluating existing models on it.

Methods

The authors followed a methodology similar to Raganato et al. (2023) to construct their Ukrainian Visual-WSD benchmark. They began by identifying homonyms in the Ukrainian language, primarily using a digitized dictionary of Ukrainian homonyms. This dictionary was refined through Optical Character Recognition (OCR) and then further curated by domain experts who selected nouns with high usage frequency and complete grammatical forms.

For each homonym, the benchmark entry consists of the target word, one or more "trigger words" to help disambiguate the intended meaning, and ten unique images. One image serves as the "positive sample," representing the correct meaning, while the other nine are "negative samples." These negative samples are categorized into:

  • Alternative interpretations of the target word.
  • Similar words within the same domain.
  • Randomly selected concepts.

Positive samples were primarily sourced from the main image of the corresponding Ukrainian Wikipedia article. When Wikipedia lacked suitable images, domain experts provided them. Negative samples were generated by finding related Wikipedia articles and extracting images. The authors also employed domain experts to ensure the trigger words effectively guided the disambiguation process without making the image meaning obvious in isolation.

Datasets

The primary dataset introduced in this paper is the Ukrainian Visual Word Sense Disambiguation (U-VWSD) benchmark. This benchmark consists of 87 homonyms, each with a target word, trigger words, and ten images (one positive, nine negative). The authors plan to expand this benchmark in the future.

For evaluation, they utilized the SemEval-2023 Task 1 Visual-WSD datasets for English, Italian, and Farsi as a baseline for comparison.

Key Insights and Ideas

  • Multilingual MLLMs struggle with Ukrainian Visual-WSD: The most striking insight is the significant performance gap observed between Ukrainian and English on the Visual-WSD task, even for models designed to be multilingual. This suggests that current approaches to multilingual MLLM training and adaptation are not sufficiently robust for low-resource languages in multimodal contexts.
  • Homonymy is a critical challenge for MLLMs: The paper highlights how homonyms, words with the same spelling but different meanings, pose a substantial challenge for MLLMs, especially when visual context is involved. This is exacerbated in languages with rich homonymy and less available training data.
  • The need for language-specific multimodal resources: The lack of dedicated multimodal benchmarks for languages like Ukrainian hinders the development and evaluation of truly inclusive AI systems. The U-VWSD benchmark is a crucial step towards addressing this.

Key Results

The paper reports that all eight tested multilingual and multimodal large language models performed worse than the zero-shot CLIP-based baseline model used for the English Visual-WSD task. Specifically, the best-performing model on the U-VWSD benchmark achieved a HIT@1 score of 42.78%, while the English baseline achieved 60.48%. This demonstrates a substantial performance disparity. The authors also note that models like LLaVA-1.5, which are based on visual instruction tuning, showed particularly low performance on the Ukrainian benchmark.

Limitations and Future Work

Limitations: * Limited homonym set: The current benchmark is based on a limited set of 87 homonyms, primarily focusing on nouns with high usage frequency. The authors acknowledge that many other undocumented homonyms exist in Ukrainian neo-lexicography that could significantly impact model evaluation. * Focus on nouns: The benchmark currently only includes nouns, which might not fully capture the complexities of homonymy across all word classes.

Future Work: * Expand the homonym list: The authors plan to include homonyms that are part of modern Ukrainian speech but may not be formally documented in existing dictionaries. This includes terms from specialized domains or informal language. * Publish benchmark and API: The U-VWSD benchmark and its associated homonym definitions will be made publicly available online with an API for broader research access. * Integrate into existing benchmarks: The goal is to integrate the Ukrainian benchmark into existing multilingual and multimodal LLM benchmarks to facilitate cross-language research. * Address other word classes: Future work could involve expanding the benchmark to include homonyms from other parts of speech.

Summary updated

BibTeX
@inproceedings{laba2024ukrainian,
  author = {Yurii Laba and Yaryna Mohytych and Ivanna Rohulia and Halyna Kyryleyza and Hanna Dydyk-Meush and Oles Dobosevych and Rostyslav Hryniv},
  title = {{Ukrainian Visual Word Sense Disambiguation Benchmark}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure