Image Description using Visual Dependency Representations

Desmond Elliott, Frank KellerEMNLP 2013 · OpenReview

Abstract: Describing the main event of an image involves identifying the objects depicted and predicting the relationships between them. Previous approaches have represented images as unstructured bags of regions, which makes it difficult to accurately predict meaningful relationships between regions. In this paper, we introduce visual dependency representations to capture the relationships between the objects in an image, and hypothesize that this representation can improve image description. We test this hypothesis using a new data set of region-annotated images, associated with visual dependency representations and gold-standard descriptions. We describe two template-based description generation models that operate over visual dependency representations. In an image description task, we find that these models outperform approaches that rely on object proximity or corpus information to generate descriptions on both automatic measures and on human judgements.

Introduction

In 2013, the state-of-the-art in image description often treated visual scenes as "unstructured bags of regions." While effective for simple object labeling, this approach fails to capture the "main event"—the functional and spatial interactions that define what is actually happening. For instance, a "man" and a "bike" occupy the same general space whether the man is riding, repairing, or standing next to the bicycle. This paper addresses this limitation by introducing Visual Dependency Representations (VDRs), a structured format inspired by linguistic dependency grammars.

Visual Dependency Representations

The core innovation is the representation of an image as a directed acyclic graph. In this structure, a "central actor" (typically the person or object performing the primary action) is attached to a ROOT node. Other objects are then hierarchically attached based on their spatial and functional relationships.

Visual Dependency Grammar

The authors defined a grammar consisting of eight spatial relations: on, surrounds, above, below, infront, behind, left, and right. These relations are determined by geometric properties such as pixel overlap and relative angles between object regions.

Geometric definition of the 'surrounds' relation in the visual dependency grammar.

Mapping Vision to Language

The VDR provides a scaffold for language generation. By mapping visual edges to linguistic dependencies, the system can infer verbs and prepositions that describe the scene accurately.

The complete pipeline: (a) segmented image regions, (b) human-written descriptions, and (c) the mapping between visual and linguistic dependency trees.

Generation Models

The authors evaluated four template-based models to test the efficacy of VDRs:

  • PROXIMITY: A baseline relating objects based on physical nearness.
  • CORPUS: Uses external text (English Gigaword) to predict likely verbs between objects.
  • STRUCTURE: Uses VDRs but is restricted to spatial terms (e.g., "The man is above the bike").
  • PARALLEL: The most advanced model, which learns to map VDR edges to specific action verbs (like "riding") by training on a parallel corpus of images and descriptions.

Experimental Results

The study utilized the PASCAL VOC 2011 action recognition dataset, augmented with 7,272 human-written descriptions and 1,023 gold-standard VDRs. The researchers found that while "bag-of-regions" models are decent at describing the background scene, they are fundamentally ill-equipped to describe the action.

The PARALLEL model significantly outperformed all baselines. On the BLEU-4 metric, it achieved a score of 2.9, compared to just 0.4 for the PROXIMITY model. Human evaluations mirrored these results; for action description accuracy, the PARALLEL model scored 3.4 on a 5-point scale, while the next best non-VDR model (CORPUS) scored only 2.2.

Conclusion

This research demonstrates a deep structural symmetry between visual perception and linguistic structure. By treating an image as a parseable tree rather than a bag of objects, the authors showed that algorithms can move beyond simple labeling toward true scene understanding. While limited by template-based generation and the current need for region annotations, the VDR framework provides a robust foundation for structured visual-linguistic modeling.

Summary updated

Abstract

Describing the main event of an image involves identifying the objects depicted and predicting the relationships between them. Previous approaches have represented images as unstructured bags of regions, which makes it difficult to accurately predict meaningful relationships between regions. In this paper, we introduce visual dependency representations to capture the relationships between the objects in an image, and hypothesize that this representation can improve image description. We test this hypothesis using a new data set of region-annotated images, associated with visual dependency representations and gold-standard descriptions. We describe two template-based description generation models that operate over visual dependency representations. In an image description task, we find that these models outperform approaches that rely on object proximity or corpus information to generate descriptions on both automatic measures and on human judgements.
BibTeX
@inproceedings{elliott2013image,
  author = {Desmond Elliott and Frank Keller},
  title = {{Image Description using Visual Dependency Representations}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2013},
  url = {https://openreview.net/forum?id=HJE3uzzO-H},
}
Figure