Neural Storyboard Artist: Visualizing Stories with Coherent Image Sequences

Shizhe Chen, Bei Liu, Jianlong Fu, Ruihua Song, Qin Jin, Pingping Lin, Xiaoyu Qi, Chunting Wang, Jin Zhou

Abstract

A storyboard is a sequence of images that illustrates a multi-sentence story, serving as a critical step in creative production. We introduce the Neural Storyboard Artist, an automatic framework designed to facilitate this process. Our "Inspire-and-Create" approach utilizes a story-to-image retriever to select relevant cinematic images for inspiration, followed by a storyboard creator that refines these images for better relevancy and visual consistency. The retriever uses contextual information and dense visual-semantic matching, while the creator employs region segmentation, style unification, and character substitution. Experiments on VIST and a new GraphMovie dataset demonstrate that our model produces high-quality, coherent storyboards that outperform existing baselines.

Introduction

Creating a storyboard is a labor-intensive process that requires both artistic skill and narrative intuition. While generative models like GANs offer flexibility, they often struggle with the complexity of "in-the-wild" scenes, resulting in blurry or distorted images. Conversely, traditional retrieval-based methods provide high-quality cinematic images but often fail to maintain narrative coherence or visual consistency across a sequence. The Neural Storyboard Artist addresses these challenges by bridging the gap between high-quality retrieval and flexible rendering.

The Inspire-and-Create Framework

The proposed framework consists of two primary stages: the "Inspire" stage for context-aware image retrieval and the "Create" stage for visual refinement.

Overview of the Inspire-and-Create framework, showing the flow from story text to retrieved images and finally to the rendered storyboard.

1. The "Inspire" Stage: Contextual-Aware Dense Matching

The Contextual-Aware Dense Matching (CADM) retriever moves beyond isolated sentence matching. It employs a hierarchical attention mechanism to incorporate surrounding story context, which is crucial for resolving pronouns and maintaining thematic continuity.

The architecture of the Contextual-aware Sentence Encoder, highlighting the hierarchical layers from words to sentences to the joint story representation.
  • Dense Matching: The system grounds specific words to corresponding image regions, ensuring that the visual content accurately reflects the textual details.
  • One-to-Many Coverage: A greedy decoding algorithm allows for the retrieval of multiple complementary images for complex sentences, ensuring comprehensive coverage of the narrative.

2. The "Create" Stage: Storyboard Creator

After retrieval, the system applies three rendering steps to unify the visual sequence:

  • Relevant Region Segmentation: By fusing dense matching grounding with Mask R-CNN, the system isolates story-relevant objects and removes distracting backgrounds.
  • Style Unification: To ensure a consistent aesthetic, CartoonGAN is used to apply a uniform artistic style across the entire sequence.
  • 3D Model Substitution: To maintain character consistency, 3D character templates are overlaid on retrieved images, ensuring the protagonist's appearance remains stable throughout the story.

Datasets and Evaluation

The model was pre-trained on MSCOCO and evaluated on the Visual Storytelling (VIST) dataset. Additionally, the authors introduced GraphMovie, a large-scale dataset of 3 million sentence-image pairs crawled from movie explanatory sites, providing a rich pool of cinematic candidates.

On the VIST dataset, the CADM model achieved a Recall@100 of 45.53%, a significant improvement over the state-of-the-art CNSI baseline (4.5%). Human evaluations further supported these findings, with CADM storyboards rated as "good" 52.2% of the time, compared to 38.4% for non-contextual models.

Conclusion and Future Work

The Neural Storyboard Artist demonstrates that leveraging inter-sentence context and high-quality retrieval foundations can produce professional-grade storyboards. Future work will focus on automating the character substitution process and explicitly modeling cinematic camera angles to further enhance the "director's vision" in automated storyboarding.

Summary updated

Abstract

A storyboard is a sequence of images to illustrate a story containing multiple sentences, which has been a key process to create different story products. In this paper, we tackle a new multimedia task of automatic storyboard creation to facilitate this process and inspire human artists. Inspired by the fact that our understanding of languages is based on our past experience, we propose a novel inspire-and-create framework with a story-to-image retriever that selects relevant cinematic images for inspiration and a storyboard creator that further refines and renders images to improve the relevancy and visual consistency. The proposed retriever dynamically employs contextual information in the story with hierarchical attentions and applies dense visual-semantic matching to accurately retrieve and ground images. The creator then employs three rendering steps to increase the flexibility of retrieved images, which include erasing irrelevant regions, unifying styles of images and substituting consistent characters. We carry out extensive experiments on both in-domain and out-of-domain visual story datasets. The proposed model achieves better quantitative performance than the state-of-the-art baselines for storyboard creation. Qualitative visualizations and user studies further verify that our approach can create high-quality storyboards even for stories in the wild.
BibTeX
@inproceedings{chen2019neural,
  author = {Shizhe Chen and Bei Liu and Jianlong Fu and Ruihua Song and Qin Jin and Pingping Lin and Xiaoyu Qi and Chunting Wang and Jin Zhou},
  title = {{Neural Storyboard Artist: Visualizing Stories with Coherent Image Sequences}},
  booktitle = {ACM Multimedia},
  year = {2019},
  doi = {10.1145/3343031.3350571},
}
Figure