Abstract
Computational visual storytelling produces a textual description of events and interpretations depicted in a sequence of images. These texts are made possible by advances and cross-disciplinary approaches in natural language processing, generation, and computer vision. We define a computational creative visual storytelling as one with the ability to alter the telling of a story along three aspects: to speak about different environments, to produce variations based on narrative goals, and to adapt the narrative to the audience. These aspects of creative storytelling and their effect on the narrative have yet to be explored in visual storytelling. This paper presents a pipeline of task-modules, Object Identification, Single-Image Inferencing, and Multi-Image Narration, that serve as a preliminary design for building a creative visual storyteller. We have piloted this design for a sequence of images in an annotation task. We present and analyze the collected corpus and describe plans towards automation.
Introduction: Beyond Literal Captioning
Most computer vision systems excel at literal description (e.g., "a person sitting at a desk"), but they struggle with visual storytelling, which requires drawing connections between images, inferring intent, and adapting to an audience. Current models are often trained on high-quality, "canonical" datasets where objects are clear and contexts are standard. However, these models frequently fail in "low-resourced" or atypical domains—such as images from a robot surveying a messy construction site or a blurry camera used by a child. This paper proposes a modular three-stage pipeline designed to move AI from literal image captioning toward "creative" storytelling by "reading between the lines" to create a cohesive narrative.
The Proposed Pipeline
The authors propose a modular pipeline that mimics human cognitive steps in storytelling, moving from raw perception to high-level synthesis.
- Task-Module 1: Object Identification (T1): Answers "What is here?" by identifying objects and their attributes (color, location, orientation).
- Task-Module 2: Single-Image Inferencing (T2): Answers "What happens here?" by moving from literal labels to interpretations (e.g., seeing a "crockpot" and "cereal" and inferring a "kitchen area").
- Task-Module 3: Multi-Image Narration (T3): Answers "What has happened so far?" by weaving inferences from a sequence of images into a single story, maintaining temporal consistency.
Pilot Study and Dataset
To validate this framework, a pilot experiment was conducted with 10 human annotators acting as the "modules." The dataset consisted of a sequence of three images taken from a ground robot in an unfamiliar, messy indoor environment. These images were intentionally low-quality and atypical to force annotators to use creative inference rather than relying on obvious patterns.
Key Insights and Results
The pilot study revealed several critical aspects of how humans construct narratives from visual data:
- Selective Attention: Annotators filtered observations based on the story they were building. For instance, if they perceived the room as an office, they focused on the computer; if they saw it as a squatting site, they focused on the suitcase.
- Narrative Shifts: New information in later images often forced a re-evaluation of previous ones. Seeing a kitchen-like setup in Image 3 often changed the interpretation of Image 1 from an "office" to a "temporary living space."
- Uncertainty as a Creative Tool: Annotators frequently used "hedging" language (e.g., "maybe," "looks like"). This suggests that automated storytellers should preserve uncertainty rather than forcing high-confidence labels.
- Interpretive Variance: Narratives ranged wildly from "military briefing room" to "temporary home for a traveler," despite seeing the same images.
The results validated the modularity of the design: while T2 (inference) depends on T1 (objects), T3 (the story) is where creative synthesis happens, often overriding or refining initial inferences.
Future Work
The authors plan to scale this research using web-based crowdsourcing to collect a larger corpus. Future work will focus on the actual automation of these modules and the development of new evaluation metrics that prioritize "plausibility" and "creativity" over simple word overlap (like BLEU or ROUGE). Additionally, the system will be tested for audience adaptation, prompting narratives for specific targets like "concise news" versus "humorous entertainment."
Summary updated