Abstract
Verb Sense Disambiguation (VSD) is a core task in Natural Language Processing, traditionally focused on textual context. This work explores Visual Verb Sense Disambiguation (VVSD), where the correct sense of a verb is determined by an associated image. Annotating multimodal datasets for VVSD is labor-intensive and requires specialized skills. We propose a transductive semi-supervised learning (SSL) approach that significantly reduces the need for labeled data. By framing the task as a Graph Transduction Game (GTG), our model achieves state-of-the-art performance on the VerSe dataset using only a small fraction of labeled samples.
Introduction
Verbs often possess multiple meanings that are context-dependent. For instance, the verb "run" takes on distinct senses when applied to a person, a machine, or a political campaign. While traditional Word Sense Disambiguation relies on surrounding text, VVSD utilizes visual information to resolve ambiguity. The primary challenge in this field is the "annotation bottleneck": labeling <image, verb> pairs is cognitively demanding and expensive.
This paper introduces TVVSD (Transductive Visual Verb Sense Disambiguation), a method designed to bridge the gap between purely unsupervised methods and data-hungry supervised models. By leveraging the underlying structure of unlabeled data alongside a few labeled examples, TVVSD minimizes human annotation effort while maximizing accuracy.
Methodology
The TVVSD framework follows a four-stage pipeline to resolve verb senses:
- Feature Extraction: Multimodal representations are generated for each <image, verb> pair. Visual features are extracted using a pre-trained VGG16 network, while textual features are derived from Word2Vec embeddings of image captions or detected object labels.
- Graph Construction: The dataset is represented as a weighted graph where nodes are <image, verb> pairs and edges represent cosine similarity between their multimodal features.
- Initial Assignment: Labeled nodes are fixed to their ground-truth senses. Unlabeled nodes are initialized with a uniform probability distribution.
- Transductive Inference: The disambiguation process is modeled as a non-cooperative game. Nodes act as players choosing a sense (strategy) to maximize compatibility with their neighbors. This is solved via replicator dynamics, an iterative process that allows consistent labeling hypotheses to propagate through the graph until a stable consensus is reached.
Experimental Setup
The model was evaluated on the VerSe dataset, which contains 3,510 <image, verb> pairs across 90 verbs and 163 senses. Verbs are categorized into "motion" (e.g., crawl) and "non-motion" (e.g., believe). A subset, VerSe-19verbs, was used for more rigorous comparison against supervised baselines. The authors tested two textual context scenarios: GOLD (human-annotated) and PRED (machine-generated) labels/captions.
Results and Discussion
TVVSD demonstrates remarkable efficiency, outperforming existing state-of-the-art methods with minimal supervision. A key finding is that providing just two labeled samples per sense allows the model to surpass previous unsupervised benchmarks.
As shown in the results, the model reaches a performance plateau with only 6-8 labeled points per sense, suggesting that massive annotation efforts yield diminishing returns for this architecture. Furthermore, the model consistently beats the "Most Frequent Sense" (MFS) heuristic, a common and difficult baseline in WSD tasks.
The integration of multimodal features acts as a safety net: visual features excel for motion verbs, while textual features (captions/objects) provide essential context for abstract or non-motion verbs.
Conclusion
This work presents a robust transductive approach to Visual Verb Sense Disambiguation. By utilizing Graph Transduction Games and replicator dynamics, TVVSD achieves high accuracy with very few labeled examples, effectively addressing the annotation bottleneck. Future work will explore more sophisticated multimodal fusion techniques and apply this framework to related tasks such as Action Recognition and Human-Object Interaction.
Summary updated