Transductive Visual Verb Sense Disambiguation

Sebastiano Vascon, Sinem Aslan, Gianluca Bigaglia, Lorenzo Giudice, Marcello Pelillo

TL;DR

This paper introduces a transductive semi-supervised learning approach to Visual Verb Sense Disambiguation (VVSD)—the task of determining a verb's meaning based on an accompanying image rather than a sentence. By using a graph-based "game-theoretic" model to propagate labels, the authors achieve state-of-the-art results using only a tiny fraction of labeled data, proving that we don't need massive human-annotated datasets to teach AI the nuances of action and context.

What Problem Does This Paper Address?

In Natural Language Processing (NLP), Word Sense Disambiguation (WSD) is a classic challenge: does "bank" mean a financial institution or the side of a river? While usually solved using surrounding text, the problem becomes much harder in a multimodal context.

Visual Verb Sense Disambiguation (VVSD) asks: can we determine the sense of a verb (like "run") when the only context provided is an image? For example, "run" could mean a person sprinting, a washing machine operating, or a road extending across a landscape.

The primary bottleneck for VVSD is data annotation. It is incredibly expensive and time-consuming for humans to label <image, verb> pairs because it requires both visual and linguistic precision. Previous attempts relied on either unsupervised methods (which struggle with accuracy) or fully supervised methods (which require massive amounts of data). The authors tackle this by asking: How can we achieve high accuracy with minimal human labeling?

Methods

The authors propose a model called TVVSD (Transductive Visual Verb Sense Disambiguation). The core innovation is the use of Transductive Semi-Supervised Learning (SSL). Unlike "inductive" learning, which builds a general rule to apply to new data, "transductive" learning looks at the specific set of unlabeled data points and uses their relationships to propagate information from a few labeled examples.

The pipeline consists of four main steps:

  1. Feature Extraction: They extract visual features using a pre-trained VGG16 network and textual features (from image captions or object labels) using Word2Vec.
  2. Graph Construction: They build a graph where every node is an <image, verb> pair. Edges between nodes represent the similarity between their features (visual, textual, or a combination).
  3. Initial Assignment: A small number of nodes are given "ground truth" labels. Unlabeled nodes are initialized with a uniform probability across all possible senses.
  4. Graph Transduction Games (GTG): This is the "secret sauce." The authors treat disambiguation as a non-cooperative game. Nodes "play" against each other, and labels propagate through replicator dynamics—a process inspired by evolutionary biology where "better" (more consistent) labeling hypotheses survive and spread across the graph until the system reaches a stable consensus.

Datasets

The study utilizes the VerSe dataset, which is currently the only well-curated benchmark for this task.

  • VerSe: Contains 3,510 <image, verb> pairs covering 90 verbs and 163 possible senses. Verbs are split into "motion" (e.g., "jump") and "non-motion" (e.g., "keep").
  • VerSe-19verbs: A subset of VerSe containing verbs with at least 20 images and two senses, used for more rigorous comparison against supervised baselines.

Key Insights and Ideas

  • The Power of Two: One of the most striking findings is that using just two labeled samples per sense is often enough to outperform the previous unsupervised state-of-the-art.
  • Motion vs. Non-Motion: Motion verbs are significantly easier to disambiguate. This is because motion usually involves specific interactions between entities (e.g., a person and a bicycle for the verb "ride"), which creates distinct visual and textual clusters.
  • Logarithmic Growth: Accuracy follows a logarithmic trend. The most significant performance gains happen when moving from 1 to 8 labeled samples; after that, the benefits of adding more labels begin to plateau.
  • Textual Noise: Interestingly, for non-motion verbs, adding too many labeled points using textual features can actually decrease performance slightly. This suggests that high-density labeling can sometimes introduce noise if the textual descriptions are too similar across different senses.

Key Results

The TVVSD model outperformed existing methods by a "large margin" across almost all configurations:

  • Efficiency: On the VerSe dataset, using 20 labels per sense, the model achieved ~97% accuracy for motion verbs using "GOLD" (human-verified) textual annotations, compared to the ~54% achieved by previous unsupervised methods.
  • SOTA Comparison: On the VerSe-19verbs subset, the transductive approach beat the supervised baseline (which uses 80% of the data for training) while using significantly fewer labeled examples.
  • Multimodality: While combining text and images (CNN+Text) generally helped, the model was surprisingly robust even when using only unimodal features (just text or just images), thanks to the strength of the graph-based propagation.

Limitations and Future Work

  • Feature Dependency: The model's performance is heavily tied to the quality of the initial embeddings. The authors used standard VGG16 and Word2Vec; they suggest that more modern architectures (like CLIP or Transformer-based visual features) could push these results even further.
  • Non-Motion Complexity: Disambiguating non-motion verbs remains a tougher nut to crack, as the visual cues are often more subtle than those in action-oriented motion verbs.
  • Future Directions: The authors suggest exploring more sophisticated feature embeddings, such as "ImgObjLoc," which specifically encodes the spatial location of objects within an image to better capture context.

Summary updated

Abstract

Verb Sense Disambiguation is a well-known task in NLP, the aim is to find the correct sense of a verb in a sentence. Recently, this problem has been extended in a multimodal scenario, by exploiting both textual and visual features of ambiguous verbs leading to a new problem, the Visual Verb Sense Disambiguation (VVSD). Here, the sense of a verb is assigned considering the content of an image paired with it rather than a sentence in which the verb appears. Annotating a dataset for this task is more complex than textual disambiguation, because assigning the correct sense to a pair of <image, verb> requires both non-trivial linguistic and visual skills. In this work, differently from the literature, the VVSD task will be performed in a transductive semi-supervised learning (SSL) setting, in which only a small amount of labeled information is required, reducing tremendously the need for annotated data. The disambiguation process is based on a graph-based label propagation method which takes into account mono or multimodal representations for <image, verb> pairs. Experiments have been carried out on the recently published dataset VerSe, the only available dataset for this task. The achieved results outperform the current state-of-the-art by a large margin while using only a small fraction of labeled samples per sense <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
BibTeX
@inproceedings{vascon2021transductive,
  author = {Sebastiano Vascon and Sinem Aslan and Gianluca Bigaglia and Lorenzo Giudice and Marcello Pelillo},
  title = {{Transductive Visual Verb Sense Disambiguation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=yGJpWmJoFn},
}

Loading related content...

Figure