Learning to Interpret and Describe Abstract Scenes

Luis Gilberto Mateos Ortiz, Clemens Wolff, Mirella Lapata

Introduction

Describing a visual scene requires two distinct capabilities: identifying objects and their spatial arrangements (computer vision) and generating a coherent narrative (natural language generation). In 2015, the "noise" of imperfect object detection often obscured whether captioning errors stemmed from vision or language failures. This research isolates the language grounding problem by using abstract clipart scenes, allowing the authors to focus on how a machine decides what is important to mention and how to weave those observations into fluent sentences.

Visual Dependency Grammar (VDG)

To represent scenes structurally, the authors moved away from simple "bags-of-objects." Instead, they modeled scenes using Visual Dependency Grammar (VDG), which characterizes a scene as a set of binary spatial relations.

Definitions of the spatial relations used to build the Visual Dependency Grammar.

These relations (e.g., above, below, near) provide a structured "clean room" environment. By using clipart metadata, the researchers could bypass the errors of raw pixel analysis and focus on the logic of content selection and surface realization.

The Translation Pipeline

The core innovation of this work is treating image captioning as a translation problem. The pipeline consists of two stages:

  1. Content Selection via Integer Linear Programming (ILP): Since humans only describe "interesting" interactions, the model uses ILP to select relevant object pairs based on co-occurrence frequency and physical proximity.
  2. Surface Realization via Statistical Machine Translation (SMT): The selected relations are treated as a source language and "translated" into English.
Mapping an abstract scene to its structured Visual Dependency Grammar (VDG) representation.

Using the Moses SMT engine, the model converts strings like hb0.10s.png close_F same t.4s.png into natural sentences such as "Mike is kicking the soccer ball." This "Vision-as-Language" metaphor allows the system to leverage established linguistic translation techniques for visual data.

Key Results and Insights

The SMT-based approach significantly outperformed early multimodal neural networks of the era. On the Abstract Scenes Dataset—which contains over 10,000 images and 60,000 human descriptions—the model achieved a 43.70 BLEU score. This was a substantial improvement over the 12.30 BLEU score of contemporary multimodal log-bilinear models.

Human evaluations confirmed these findings: judges preferred the SMT model's descriptions in 53% of cases (excluding human gold standards), and 75.5% of the top-ranked descriptions were deemed both accurate and relevant.

Conclusion

By utilizing abstract scenes, this research demonstrated that structural information—understanding "who is doing what to whom"—is more critical for high-quality captioning than raw pixel features. While the "clipart gap" remains a challenge for real-world application, the study highlights that effective content selection is non-trivial; a model must learn a hierarchy of importance to distinguish between a primary action and background elements like the sun. Future work aims to bridge these structural insights with the visual complexity of real-world photography.

Summary updated

Abstract

Given a (static) scene, a human can effortlessly describe what is going on (who is doing what to whom, how, and why). The process requires knowledge about the world, how it is perceived, and described. In this paper we study the problem of interpreting and verbalizing visual information using abstract scenes created from collections of clip art images. We propose a model inspired by machine translation operating over a large parallel corpus of visual relations and linguistic descriptions. We demonstrate that this approach produces human-like scene descriptions which are both fluent and relevant, outperforming a number of competitive alternatives based on templates, sentence-based retrieval, and a multimodal neural language model.
BibTeX
@inproceedings{ortiz2015learning,
  author = {Luis Gilberto Mateos Ortiz and Clemens Wolff and Mirella Lapata},
  title = {{Learning to Interpret and Describe Abstract Scenes}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2015},
  doi = {10.3115/v1/n15-1174},
}
Figure