TAM of SCNU at SemEval-2023 Task 1: FCLL: A Fine-grained Contrastive Language-Image Learning Model for Cross-language Visual Word Sense Disambiguation

Qihao Yang, Yong Li, Xuelin Wang, Shunhao Li, Tianyong HaoACL 2023

TL;DR

This paper introduces FCLL, a novel fine-grained contrastive learning model for cross-language Visual Word Sense Disambiguation (WSD). By learning nuanced image-text relationships and enriching contextual information, FCLL achieved state-of-the-art results on the SemEval-2023 Task 1, demonstrating its effectiveness in understanding complex language-vision semantics.

What Problem Does This Paper Address?

Word polysemy, the phenomenon of words having multiple meanings, is a long-standing challenge in Natural Language Processing (NLP). Word Sense Disambiguation (WSD) aims to identify the correct meaning of a word in a given context. While traditional WSD focuses on textual context, Visual WSD extends this by requiring models to link ambiguous words to relevant images. This task is particularly challenging due to:

  • Limited Context: Textual snippets are often short, making it hard to infer the precise meaning of an ambiguous word.
  • Cross-linguistic Nuances: Words can have different senses or connotations across languages, and syntactic differences further complicate translation and understanding.
  • Fine-grained Image-Text Alignment: Effectively connecting abstract word senses to concrete visual representations requires a deep understanding of subtle image-text relationships.

Existing models often struggle with these challenges, particularly in cross-language scenarios and when dealing with the fine-grained nature of visual WSD.

Methods

The core of this work is the FCLL (Fine-grained Contrastive Language-Image Learning) model. Its key components include:

  • Fine-grained Text Encoder: This module leverages CLIP's architecture to learn semantic consistency between senses and images. It uses an "image queue" to store retrieved images, enabling contrastive learning from the text-inferred image perspective.
  • Fine-grained Image Encoder: Similar to the text encoder, this module uses CLIP but focuses on learning semantic consistency between images and senses. It employs a "text queue" to store retrieved senses, facilitating contrastive learning from the image-inferred text perspective.
  • Fine-grained Image-Text Matching Predictor: A transformer-based module that fuses multimodal features from words, senses, and images, further enhancing contrastive learning through feature fusion.
  • Word Sense Auto-complementing Module: This novel strategy addresses the issue of limited textual context. It generates "concepts" by combining the ambiguous target word with parts of its full sense (after removing stop words). An exclusive sense encoder then complements the English contextual information to these concepts, improving the model's understanding of phrases.
  • Contrastive Losses: The model utilizes three contrastive losses: Sense-to-Image Contrastive (SIC), Image-to-Sense Contrastive (ISC), and Text-concept-Text full Contrastive (TTC). These losses help align representations in a shared space.
  • Image-Text Matching Loss: A cross-entropy loss for binary classification, similar to BLIP, to further refine the model's ability to match images and text.

Datasets

  • SemEval-2023 Task 1 Datasets: Used for training and evaluation. These datasets contain ambiguous target words and candidate images across English, Farsi, and Italian. The paper notes significant imbalance in the training set proportions across languages.
  • V-WSD KB (Visual Word Sense Disambiguation Knowledge Base): A newly constructed multimodal-multilingual knowledge base. It contains 12,956 ambiguous target words, 20,904 English senses, and 97,267 relevant images across English, Farsi, and Italian. This KB was built using BabelNet and aims to address the limitations of existing datasets (low data volume, monolingual focus, lack of context).

Key Insights and Ideas

  • Fine-grained alignment is crucial: The success of FCLL highlights the importance of learning subtle, fine-grained relationships between language and vision, rather than just coarse-level correspondences.
  • Contextual enrichment is key for short texts: The "Word Sense Auto-complementing" module demonstrates that augmenting limited textual context with generated concepts can significantly improve performance, especially in cross-lingual settings.
  • Multimodal-multilingual knowledge is essential: The creation and use of the V-WSD KB underscore the need for comprehensive, multilingual resources to advance cross-language visual WSD research.
  • Contrastive learning excels at alignment: The paper effectively leverages contrastive learning principles to align representations from different modalities and languages.

Key Results

FCLL achieved state-of-the-art performance on the SemEval-2023 Task 1.

  • Overall Evaluation: FCLL ranked first with an average H@1 of 72.56% and an average MRR of 82.22%.
  • Cross-language Flexibility: The model demonstrated strong performance across English, Farsi, and Italian, outperforming other strong baselines like CLIP, BLIP, and FLAVA, even when they were used in a zero-shot setting on translated test data.
  • Ablation Study: The study revealed that the fine-grained text encoder and the sense auto-complementing module were the most critical components for FCLL's success.

Limitations and Future Work

  • Limitations: The paper notes that the SemEval-2023 Task 1 datasets suffer from low data volume, are primarily monolingual (English), and lack sufficient contextual information. While V-WSD KB addresses some of these, the inherent limitations of the original task datasets remain.
  • Future Work: The authors suggest that their FCLL model and the V-WSD KB can serve as a foundation for future research in fine-grained language-image understanding and cross-language visual WSD. The release of their source code and knowledge base aims to facilitate this.

Summary updated

BibTeX
@inproceedings{yang2023tam,
  author = {Qihao Yang and Yong Li and Xuelin Wang and Shunhao Li and Tianyong Hao},
  title = {{TAM of SCNU at SemEval-2023 Task 1: FCLL: A Fine-grained Contrastive Language-Image Learning Model for Cross-language Visual Word Sense Disambiguation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure