Transductive Visual Verb Sense Disambiguation

Sebastiano Vascon, Sinem Aslan, Gianluca Bigaglia, Lorenzo Giudice, Marcello Pelillo

Abstract

Verb Sense Disambiguation (VSD) is a core task in Natural Language Processing, traditionally focused on textual context. This work explores Visual Verb Sense Disambiguation (VVSD), where the correct sense of a verb is determined by an associated image. Annotating multimodal datasets for VVSD is labor-intensive and requires specialized skills. We propose a transductive semi-supervised learning (SSL) approach that significantly reduces the need for labeled data. By framing the task as a Graph Transduction Game (GTG), our model achieves state-of-the-art performance on the VerSe dataset using only a small fraction of labeled samples.

Introduction

Verbs often possess multiple meanings that are context-dependent. For instance, the verb "run" takes on distinct senses when applied to a person, a machine, or a political campaign. While traditional Word Sense Disambiguation relies on surrounding text, VVSD utilizes visual information to resolve ambiguity. The primary challenge in this field is the "annotation bottleneck": labeling <image, verb> pairs is cognitively demanding and expensive.

This paper introduces TVVSD (Transductive Visual Verb Sense Disambiguation), a method designed to bridge the gap between purely unsupervised methods and data-hungry supervised models. By leveraging the underlying structure of unlabeled data alongside a few labeled examples, TVVSD minimizes human annotation effort while maximizing accuracy.

Methodology

The TVVSD framework follows a four-stage pipeline to resolve verb senses:

  1. Feature Extraction: Multimodal representations are generated for each <image, verb> pair. Visual features are extracted using a pre-trained VGG16 network, while textual features are derived from Word2Vec embeddings of image captions or detected object labels.
  2. Graph Construction: The dataset is represented as a weighted graph where nodes are <image, verb> pairs and edges represent cosine similarity between their multimodal features.
  3. Initial Assignment: Labeled nodes are fixed to their ground-truth senses. Unlabeled nodes are initialized with a uniform probability distribution.
  4. Transductive Inference: The disambiguation process is modeled as a non-cooperative game. Nodes act as players choosing a sense (strategy) to maximize compatibility with their neighbors. This is solved via replicator dynamics, an iterative process that allows consistent labeling hypotheses to propagate through the graph until a stable consensus is reached.
The TVVSD pipeline: from multimodal feature extraction to graph-based label propagation using replicator dynamics.

Experimental Setup

The model was evaluated on the VerSe dataset, which contains 3,510 <image, verb> pairs across 90 verbs and 163 senses. Verbs are categorized into "motion" (e.g., crawl) and "non-motion" (e.g., believe). A subset, VerSe-19verbs, was used for more rigorous comparison against supervised baselines. The authors tested two textual context scenarios: GOLD (human-annotated) and PRED (machine-generated) labels/captions.

Results and Discussion

TVVSD demonstrates remarkable efficiency, outperforming existing state-of-the-art methods with minimal supervision. A key finding is that providing just two labeled samples per sense allows the model to surpass previous unsupervised benchmarks.

Performance comparison on the VerSe-19verbs subset against supervised and unsupervised baselines, highlighting the effectiveness of the transductive approach.

As shown in the results, the model reaches a performance plateau with only 6-8 labeled points per sense, suggesting that massive annotation efforts yield diminishing returns for this architecture. Furthermore, the model consistently beats the "Most Frequent Sense" (MFS) heuristic, a common and difficult baseline in WSD tasks.

The integration of multimodal features acts as a safety net: visual features excel for motion verbs, while textual features (captions/objects) provide essential context for abstract or non-motion verbs.

Conclusion

This work presents a robust transductive approach to Visual Verb Sense Disambiguation. By utilizing Graph Transduction Games and replicator dynamics, TVVSD achieves high accuracy with very few labeled examples, effectively addressing the annotation bottleneck. Future work will explore more sophisticated multimodal fusion techniques and apply this framework to related tasks such as Action Recognition and Human-Object Interaction.

Summary updated

Abstract

Verb Sense Disambiguation is a well-known task in NLP, the aim is to find the correct sense of a verb in a sentence. Recently, this problem has been extended in a multimodal scenario, by exploiting both textual and visual features of ambiguous verbs leading to a new problem, the Visual Verb Sense Disambiguation (VVSD). Here, the sense of a verb is assigned considering the content of an image paired with it rather than a sentence in which the verb appears. Annotating a dataset for this task is more complex than textual disambiguation, because assigning the correct sense to a pair of $<$image, verb$>$ requires both non-trivial linguistic and visual skills. In this work, differently from the literature, the VVSD task will be performed in a transductive semi-supervised learning (SSL) setting, in which only a small amount of labeled information is required, reducing tremendously the need for annotated data. The disambiguation process is based on a graph-based label propagation method which takes into account mono or multimodal representations for $<$image, verb$>$ pairs. Experiments have been carried out on the recently published dataset VerSe, the only available dataset for this task. The achieved results outperform the current state-of-the-art by a large margin while using only a small fraction of labeled samples per sense. Code available: https://github.com/GiBg1aN/TVVSD.
BibTeX
@inproceedings{vascon2021transductive,
  author = {Sebastiano Vascon and Sinem Aslan and Gianluca Bigaglia and Lorenzo Giudice and Marcello Pelillo},
  title = {{Transductive Visual Verb Sense Disambiguation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=yGJpWmJoFn},
}
Figure