Visual Definition Modeling: Challenging Vision & Language Models to Define Words and Objects

Bianca Scarlini, Tommaso Pasini, Roberto NavigliAAAI 2022 · DOI

Presented at AAAI-22, this research introduces Visual Definition Modeling (VDM), a novel challenge designed to probe the depth of semantic understanding in multimodal AI. While contemporary Vision & Language (V&L) models excel at describing scenes or answering questions, VDM asks a more fundamental question: do these models truly "understand" the specific semantics of the objects they identify? The authors propose that the ultimate test of such understanding is the ability to generate precise, dictionary-style definitions grounded in visual context.

The Visual Definition Modeling Task

The VDM task requires models to generate textual definitions for words or specific image regions based on visual context. This moves beyond "coarse" tasks like image labeling or high-level captioning. For instance, while a model might label a "mouse" in a photograph, VDM tests whether it can distinguish between a biological rodent and a computer peripheral by generating a contextually appropriate definition.

Visual Definition Modeling (VDM) task example: defining a computer mouse based on visual context.

The authors define two primary settings for VDM:

  1. Word-based context: Defining a specific word (e.g., "hike") when provided with an image that illustrates the concept.
  2. Object-based context: Defining a specific visual region, indicated by a bounding box, within a larger image.

Benchmarking with DEMETER and DIONYSUS

To facilitate this research, the authors introduced two large-scale benchmarks:

  • DEMETER (DEfining Multimodally-contExtualized TERms): Comprising approximately 90,000 instances from BabelPic and ImageNet, this dataset includes both concrete objects (e.g., "airplane") and abstract concepts (e.g., "sadness"). It challenges models to link words to their visual representations.
  • DIONYSUS (DefIning Objects Narrowed bY viSUal Subregions): Built from MSCOCO with roughly 111,000 instances, this dataset focuses on specific object patches. It is specifically designed to eliminate semantic ambiguity through visual grounding.
Example of word-based context from the DEMETER dataset, illustrating the concept of a "hike".

Methodology: The Power of Pretrained Language Models

The study evaluated several architectural approaches, including single-stream models (VL-BERT) and dual-stream models (LXMERT). However, the standout performer was Visually-Enhanced BART (veBART).

Unlike models built from the ground up for multimodal tasks, veBART leverages a standard pretrained BART model—an encoder-decoder architecture optimized for text generation. The authors integrated visual features (extracted via Faster R-CNN) into BART’s input space using a simple linear projection layer. This approach highlights a significant "generation gap": standard V&L models, often pretrained on discriminative tasks, struggle with the generative demands of writing definitions.

Key Findings and Results

The experimental results revealed a clear hierarchy in performance, with veBART consistently outperforming multimodal-first architectures:

  • Performance Gap: On the DEMETER dataset, veBART achieved a BLEU score of 26.7, significantly higher than VL-BERT's 18.8. On DIONYSUS, the gap widened further, with veBART reaching 76.4 BLEU, while LXMERT (even with a BART decoder) only reached 57.8.
  • Language Foundation: The success of veBART suggests that the internal semantic structure of a large language model provides a more robust foundation for definition generation than the joint representations learned by current V&L models.
  • Importance of Pretraining: Models utilizing pretrained decoders (BART weights) vastly outperformed those trained from scratch, emphasizing that high-quality text generation is a skill primarily inherited from massive text-only pretraining.

Conclusion and Future Directions

Visual Definition Modeling provides a rigorous framework for assessing the semantic depth of AI models. The findings suggest that visual context is essential for word sense disambiguation and that leveraging the generative power of large language models is currently the most effective strategy for this task.

Future work may involve expanding VDM to other modalities, such as audio, and addressing current limitations like the reliance on fixed visual vocabularies from object detectors. Ultimately, VDM serves as a stepping stone toward models that possess a deeper, "common sense" understanding of the world they perceive.

Summary updated

BibTeX
@inproceedings{scarlini2022visual,
  author = {Bianca Scarlini and Tommaso Pasini and Roberto Navigli},
  title = {{Visual Definition Modeling: Challenging Vision \& Language Models to Define Words and Objects}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i10.21377},
}
Figure