Abstract
Word Sense Disambiguation (WSD) has traditionally been treated as two separate tasks: textual and visual. PolCLIP introduces a unified framework that bridges the modality gap by generating implicit complementary representations. By "hallucinating" missing modal information—visual vectors for text and textual representations for images—PolCLIP achieves state-of-the-art performance on both traditional textual and visual WSD benchmarks.
Introduction
Word Sense Disambiguation (WSD) is the task of determining which "sense" of a polysemous word (e.g., "bank" as a river edge vs. a financial institution) is intended in a specific context. Historically, this has been split into two isolated subtasks: Textual-WSD (matching a word in context to a dictionary definition) and Visual-WSD (matching a word in context to a representative image).
The challenge is that these tasks are data-starved in opposite ways. Textual-WSD datasets rarely have associated images, and Visual-WSD datasets lack formal sense definitions. Because of this "modality gap," models are usually specialized for one task and fail at the other. PolCLIP addresses this by creating a unified model that can handle both, even when one modality is entirely missing from the input.
Methodology
PolCLIP is built upon the CLIP (Contrastive Language-Image Pre-training) architecture, but introduces a novel image-text complementarity strategy to bridge the gap between senses and images.
Key components include:
- SIG (Sense-to-Image Generation) Module: When given a textual sense definition, this module generates an implicit visual vector. It essentially asks: "If this definition were an image, what would its CLIP embedding look like?"
- ISG (Image-to-Sense Generation) Module: Conversely, this module takes an image and generates an implicit textual representation.
- Momentum Encoders: To prevent catastrophic forgetting during training, the authors use momentum-based encoders to provide stable pseudo-targets for the generation modules.
- Cross-Attention Fusion: A shared 4-layer module that blends original unimodal features with the newly generated "complementary" features to create a semantically enriched multimodal representation.
- D-GPT (Disambiguation-GPT): During inference, a fine-tuned GPT-3.5 model generates a likely lexical definition for the target word in context, acting as a "semantic anchor."
Datasets
The authors constructed a massive disambiguation-oriented image-sense dataset by integrating SemCor (textual gold standard), VWSD-KB (multimodal knowledge base), and BabelNet/BabelPic (aligning senses with images). The resulting training set, Image-Enhanced SemCor, contains over 274,000 instances where senses and images are aligned. Evaluation was conducted using the XL-WSD framework for textual tasks and SemEval-2023 Task 1 for visual tasks.
Results and Discussion
PolCLIP set new records on both fronts, as shown in the comparative analysis below.
Key findings include:
- Textual-WSD: Achieved an 83.62% F1-score, a 2.53% improvement over the previous state-of-the-art (SACE).
- Visual-WSD: Achieved an 83.59% Hit Rate@1, a 2.22% improvement over the previous leader (FCLL).
- Vector-Space Hallucination: By generating vectors in the joint CLIP space, the model leverages the semantic power of the opposite modality without the computational overhead of full generative AI.
- Abstract Grounding: Visual-WSD showed significant improvements for non-concrete words (e.g., +5.72% for adverbs) because the textual generation module provides abstract grounding that raw images lack.
Conclusion
PolCLIP demonstrates that "hallucinating" missing modal information is a powerful strategy for unified WSD. While currently limited to English and relying on closed-source LLMs for context augmentation, it paves the way for more robust, multimodal language understanding. Future work will focus on multilingual support and exploring open-source alternatives for context enhancement.
Summary updated