AESOP: Abstract Encoding of Stories, Objects, and Pictures

Hareesh Ravi, Kushal Kafle, Scott Cohen, Jonathan Brandt, Mubbasir Kapadia

Introduction

In the realm of vision-and-language research, most models excel at factual reporting—identifying a "dog on a rug" or a "man in a kitchen." However, human intelligence is defined by the ability to weave these observations into a coherent narrative driven by causality and intent. To bridge this gap, researchers have introduced AESOP, a dataset and framework designed to move AI from mere description to narrative reasoning. By utilizing abstract clip-art scenes, AESOP strips away the noise of raw pixel processing to focus on the underlying logic of storytelling.

The AESOP Framework: Beyond Factual Reporting

Existing datasets often lack narrative coherence. In many cases, if the order of images in a sequence is swapped, humans struggle to identify the error because the "stories" are merely disjointed descriptions. AESOP addresses this by requiring models to understand the "why" behind visual changes.

An example of a three-panel story from the AESOP dataset, illustrating the causal link between dietary habits, a medical emergency, and subsequent lifestyle changes.

The authors propose a generalized story completion framework where the AI must function as either an Assistant Illustrator (generating a missing visual scene based on text) or an Assistant Writer (filling in missing text based on visuals).

Dataset and Methodology

The AESOP dataset consists of 7,062 stories totaling 21,186 abstract scenes. These were created by human workers using a palette of 158 unique objects (characters, animals, props) and various backgrounds. Unlike standard datasets, AESOP features a high frequency of "non-visible" verbs—such as worry, wonder, or decide—which are crucial for depicting internal character states.

Model Architecture

To process these stories, the system employs several key components:

  1. Abstract Representation: Scenes are encoded as sequences of clip-art tokens. Each object is defined by attributes such as ID, position ($x, y$), size, orientation, and facial expression.
  2. Multimodal Encoding: Separate Bidirectional GRUs encode visual panels and text sentences.
  3. Cross-Modal Attention: This layer allows the model to align visual elements with specific textual cues during generation.
  4. Hierarchical Decoding: The model first predicts the presence of an object before determining its specific state (e.g., "John is running").
A single abstract scene from the AESOP dataset, demonstrating the use of clip-art objects to represent complex social interactions and emotional states.

Key Insights: The "Jumble" Test

A defining feature of AESOP is its causal strength. In a "jumble test," where two panels in a story are swapped, humans correctly identified the original order 95% of the time for AESOP stories, compared to only 65.8% for the VIST dataset. This confirms that AESOP captures a much stronger narrative arc where every visual and textual element serves a causal purpose.

Results and Future Directions

While the proposed model outperformed simple baselines, it remains significantly behind human performance. The "Assistant Writer" task proved particularly challenging; while pre-trained models like GPT-2 are more fluent, they often lack the specific grounding required to reflect the visual details of the abstract scenes.

Future work aims to integrate massive pre-trained models with the logical constraints of AESOP. Additionally, the authors suggest moving toward "delta-prediction"—predicting the changes between frames rather than absolute scenes—to better mimic the creative process of human storytellers. By establishing a benchmark where the sequence of events is paramount, AESOP sets a new standard for the development of creative and causally-aware AI.

Summary updated

Abstract

Visual storytelling and story comprehension are uniquely human skills that play a central role in how we learn about and experience the world. Despite remarkable progress in recent years in synthesis of visual and textual content in isolation and learning effective joint visual-linguistic representations, existing systems still operate only at a superficial, factual level. With the goal of developing systems that are able to comprehend rich human-generated narratives, and co-create new stories, we introduce AESOP: a new dataset that captures the creative process associated with visual storytelling. Visual panels are composed of clip-art objects with specific attributes enabling a broad range of creative expression. Using AESOP, we propose foundational storytelling tasks that are generative variants of story cloze tests, to better measure the creative and causal reasoning ability required for visual storytelling. We further develop a generalized story completion framework that models stories as the co-evolution of visual and textual concepts. We benchmark the proposed approach with human baselines and evaluate using comprehensive qualitative and quantitative metrics. Our results highlight key insights related to the dataset, modelling and evaluation of visual storytelling for future research in this promising field of study.
BibTeX
@inproceedings{ravi2021aesop,
  author = {Hareesh Ravi and Kushal Kafle and Scott Cohen and Jonathan Brandt and Mubbasir Kapadia},
  title = {{AESOP: Abstract Encoding of Stories, Objects, and Pictures}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00206},
}
Figure