PolCLIP: A Unified Image-Text Word Sense Disambiguation Model via Generating Multimodal Complementary Representations

Qihao Yang, Yong Li, Xuelin Wang, Fu Lee Wang, Tianyong HaoACL 2024 · DOI

Abstract

Word Sense Disambiguation (WSD) has traditionally been treated as two separate tasks: textual and visual. PolCLIP introduces a unified framework that bridges the modality gap by generating implicit complementary representations. By "hallucinating" missing modal information—visual vectors for text and textual representations for images—PolCLIP achieves state-of-the-art performance on both traditional textual and visual WSD benchmarks.

Introduction

Word Sense Disambiguation (WSD) is the task of determining which "sense" of a polysemous word (e.g., "bank" as a river edge vs. a financial institution) is intended in a specific context. Historically, this has been split into two isolated subtasks: Textual-WSD (matching a word in context to a dictionary definition) and Visual-WSD (matching a word in context to a representative image).

The challenge is that these tasks are data-starved in opposite ways. Textual-WSD datasets rarely have associated images, and Visual-WSD datasets lack formal sense definitions. Because of this "modality gap," models are usually specialized for one task and fail at the other. PolCLIP addresses this by creating a unified model that can handle both, even when one modality is entirely missing from the input.

Methodology

PolCLIP is built upon the CLIP (Contrastive Language-Image Pre-training) architecture, but introduces a novel image-text complementarity strategy to bridge the gap between senses and images.

The PolCLIP architecture featuring SIG and ISG modules for cross-modal generation.

Key components include:

  • SIG (Sense-to-Image Generation) Module: When given a textual sense definition, this module generates an implicit visual vector. It essentially asks: "If this definition were an image, what would its CLIP embedding look like?"
  • ISG (Image-to-Sense Generation) Module: Conversely, this module takes an image and generates an implicit textual representation.
  • Momentum Encoders: To prevent catastrophic forgetting during training, the authors use momentum-based encoders to provide stable pseudo-targets for the generation modules.
  • Cross-Attention Fusion: A shared 4-layer module that blends original unimodal features with the newly generated "complementary" features to create a semantically enriched multimodal representation.
  • D-GPT (Disambiguation-GPT): During inference, a fine-tuned GPT-3.5 model generates a likely lexical definition for the target word in context, acting as a "semantic anchor."

Datasets

The authors constructed a massive disambiguation-oriented image-sense dataset by integrating SemCor (textual gold standard), VWSD-KB (multimodal knowledge base), and BabelNet/BabelPic (aligning senses with images). The resulting training set, Image-Enhanced SemCor, contains over 274,000 instances where senses and images are aligned. Evaluation was conducted using the XL-WSD framework for textual tasks and SemEval-2023 Task 1 for visual tasks.

Results and Discussion

PolCLIP set new records on both fronts, as shown in the comparative analysis below.

Performance comparison of PolCLIP against state-of-the-art models on Textual and Visual WSD benchmarks.

Key findings include:

  • Textual-WSD: Achieved an 83.62% F1-score, a 2.53% improvement over the previous state-of-the-art (SACE).
  • Visual-WSD: Achieved an 83.59% Hit Rate@1, a 2.22% improvement over the previous leader (FCLL).
  • Vector-Space Hallucination: By generating vectors in the joint CLIP space, the model leverages the semantic power of the opposite modality without the computational overhead of full generative AI.
  • Abstract Grounding: Visual-WSD showed significant improvements for non-concrete words (e.g., +5.72% for adverbs) because the textual generation module provides abstract grounding that raw images lack.

Conclusion

PolCLIP demonstrates that "hallucinating" missing modal information is a powerful strategy for unified WSD. While currently limited to English and relying on closed-source LLMs for context augmentation, it paves the way for more robust, multimodal language understanding. Future work will focus on multilingual support and exploring open-source alternatives for context enhancement.

Summary updated

BibTeX
@inproceedings{yang2024polclip,
  author = {Qihao Yang and Yong Li and Xuelin Wang and Fu Lee Wang and Tianyong Hao},
  title = {{PolCLIP: A Unified Image-Text Word Sense Disambiguation Model via Generating Multimodal Complementary Representations}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.575},
}
Figure