A Pipeline for Creative Visual Storytelling

Stephanie M. Lukin, Reginald Hobbs, Clare R. Voss

Abstract

Computational visual storytelling produces a textual description of events and interpretations depicted in a sequence of images. These texts are made possible by advances and crossdisciplinary approaches in natural language processing, generation, and computer vision. We define a computational creative visual storytelling as one with the ability to alter the telling of a story along three aspects: to speak about different environments, to produce variations based on narrative goals, and to adapt the narrative to the audience. These aspects of creative storytelling and their effect on the narrative have yet to be explored in visual storytelling. This paper presents a pipeline of taskmodules, Object Identification, Single-Image Inferencing, and Multi-Image Narration, that serve as a preliminary design for building a creative visual storyteller. We have piloted this design for a sequence of images in an annotation task. We present and analyze the collected corpus and describe plans towards automation.

Introduction

While computer vision (CV) and natural language processing (NLP) have made significant progress in literal image captioning, current systems often struggle to weave cohesive narratives. A literal caption might identify a "person at a desk," but it lacks the creative capacity to explain the context or link that image to a broader sequence. This paper addresses three critical gaps in visual storytelling research: environment bias (reliance on high-quality, canonical datasets), lack of narrative goals (the "point" of the story), and audience adaptation (tailoring style to the listener).

The Modular Storytelling Pipeline

To mimic the human cognitive process of storytelling, we propose a three-stage modular pipeline designed to transform sequences of images into cohesive stories:

  1. Task-Module 1: Object Identification (T1): This stage identifies the "building blocks" of the scene, including objects, their spatial co-locations, and attributes like color or shape.
  2. Task-Module 2: Single-Image Inferencing (T2): This module moves from "what" to "so what," interpreting the objects to answer "what is happening here?" (e.g., inferring a "kitchen" from the presence of a crockpot).
  3. Task-Module 3: Multi-Image Narration (T3): The final stage takes the inferences from T1 and T2 across a sequence of images to construct a larger narrative, answering "what has happened so far?"
The proposed three-stage pipeline for creative visual storytelling, illustrating the flow from object identification to multi-image narration.

Pilot Study and Dataset

We conducted a pilot experiment with 10 human annotators using a sequence of three images captured by a ground robot in an atypical, messy indoor environment. Unlike polished stock photos found in standard benchmarks like MS COCO, these images present challenges in quality and perspective, forcing annotators to engage in creative inference rather than simple labeling.

The sequence of three atypical, low-quality robot images used in the pilot study to evaluate the storytelling pipeline.

Key Insights and Results

Our analysis of the pilot corpus revealed several key aspects of creative storytelling:

  • Selective Attention: Annotators did not list every object; they filtered their observations based on the narrative they were forming. For instance, if an annotator viewed the scene as a "safe house," they focused on a suitcase; if they saw an "office," they focused on a calendar.
  • Uncertainty as a Tool: In low-quality environments, humans frequently use "hedging" (e.g., "maybe," "looks like"). We argue that AI should model this uncertainty rather than forcing high-confidence literal labels.
  • Inference Variability: The same set of objects led to wildly different narratives. Image 2, featuring a suitcase and a shirt, was interpreted by one person as a "storage room" and by another as a "hiding spot," demonstrating the inherent subjectivity of storytelling.

Conclusion and Future Work

The proposed modular approach allows researchers to pinpoint exactly where a story diverges. Future work will focus on scaling the corpus through crowdsourcing and automating the inferencing and narration modules using knowledge bases and commonsense reasoning engines. Additionally, we highlight the need for new evaluation metrics that balance grammatical correctness with narrative consistency and creativity, as traditional metrics like BLEU are insufficient for these tasks.

Summary updated

Abstract

Computational visual storytelling produces a textual description of events and interpretations depicted in a sequence of images. These texts are made possible by advances and crossdisciplinary approaches in natural language processing, generation, and computer vision. We define a computational creative visual storytelling as one with the ability to alter the telling of a story along three aspects: to speak about different environments, to produce variations based on narrative goals, and to adapt the narrative to the audience. These aspects of creative storytelling and their effect on the narrative have yet to be explored in visual storytelling. This paper presents a pipeline of taskmodules, Object Identification, Single-Image Inferencing, and Multi-Image Narration, that serve as a preliminary design for building a creative visual storyteller. We have piloted this design for a sequence of images in an annotation task. We present and analyze the collected corpus and describe plans towards automation.
BibTeX
@misc{lukin2018pipeline,
  author = {Stephanie M. Lukin and Reginald Hobbs and Clare R. Voss},
  title = {{A Pipeline for Creative Visual Storytelling}},
  year = {2018},
  eprint = {1807.08077},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1807.08077},
}
Figure