Neural Storyboard Artist: Visualizing Stories with Coherent Image Sequences

Shizhe Chen, Bei Liu, Jianlong Fu, Ruihua Song, Qin Jin, Pingping Lin, Xiaoyu Qi, Chunting Wang, Jin Zhou

TL;DR

Researchers have developed the Neural Storyboard Artist, a framework that transforms multi-sentence stories into coherent visual sequences. By combining a context-aware image retriever with a rendering pipeline that unifies styles and characters, the system moves beyond simple text-to-image matching to create "cinematic" storyboards that maintain narrative flow.

What Problem Does This Paper Address?

Creating a storyboard—a sequence of images used to plan movies or animations—is a labor-intensive task requiring artistic skill and narrative intuition. Automating this is difficult for two main reasons:

  1. The Quality-Flexibility Trade-off: Generative models (like GANs) are flexible but often produce blurry or distorted images for complex, "in-the-wild" scenes. Retrieval-based methods provide high-quality "cinematic" images but are often rigid, failing to account for story context or visual consistency across the sequence.
  2. Narrative Coherence: A storyboard isn't just a collection of random pictures; it needs a consistent style and the same characters appearing across different shots. Most existing models treat sentences in isolation, losing the "thread" of the story.

Methods

The authors propose an "Inspire-and-Create" framework consisting of two main stages:

1. The "Inspire" Stage: Contextual-Aware Dense Matching (CADM)

Instead of looking at one sentence at a time, the CADM retriever uses a hierarchical attention mechanism. It looks at the current sentence and the surrounding story context to understand pronouns (e.g., knowing "they" refers to "mom and daughter" mentioned earlier).

  • Dense Matching: It doesn't just match a sentence to an image; it grounds specific words to specific regions in an image.
  • One-to-Many Coverage: A greedy decoding algorithm allows the system to retrieve multiple complementary images for a single long or complex sentence, ensuring no story detail is left out.

2. The "Create" Stage: Storyboard Creator

Once images are retrieved, the system "simulates" an artist's touch through three rendering steps:

  • Relevant Region Segmentation: To remove distracting backgrounds, the system fuses the retriever’s "word-to-region" grounding with Mask R-CNN to precisely isolate and keep only the objects/scenes relevant to the story.
  • Style Unification: To prevent the storyboard from looking like a disjointed collage, it uses CartoonGAN to apply a consistent artistic style across all images.
  • 3D Model Substitution: To solve the "character consistency" problem, the authors use a semi-manual step where 3D character templates (via Autodesk Maya) are placed over the retrieved images to ensure the protagonist looks the same in every frame.

Datasets

  • MSCOCO: Used for pre-training the basic image-text matching.
  • VIST (Visual Storytelling): The primary benchmark for sequential image-to-story tasks.
  • GraphMovie (New): A large-scale dataset crawled from movie explanatory websites, containing over 3 million aligned sentence-image pairs. This provides the "cinematic" candidate pool for the retriever.

Key Insights and Ideas

  • Context is King: The paper demonstrates that inter-sentence context significantly boosts retrieval accuracy. Understanding the "flow" of the story helps the model pick images that make sense as a sequence, not just as individual frames.
  • Retrieval as a Foundation: Rather than trying to generate pixels from scratch (which is hard for complex scenes), the authors argue that retrieving a high-quality cinematic base and then modifying it is a more viable path for professional-grade storyboarding.
  • Complementary Segmentation: A clever insight was fusing "Dense Matching" (which knows what is relevant but has messy boundaries) with "Mask R-CNN" (which has perfect boundaries but doesn't know what is relevant to the story).

Key Results

  • Quantitative: On the VIST dataset, the CADM model achieved a Recall@100 of 45.53%, significantly outperforming the state-of-the-art baseline (CNSI) which sat at a mere 4.5%.
  • Qualitative: In human evaluations, the CADM-generated storyboards were rated as "good" 52.2% of the time, compared to 38.4% for models that didn't use story context.
  • Generalization: The model proved robust even for "stories in the wild," such as Chinese idioms and movie scripts that weren't in the training set.

Limitations and Future Work

  • Semi-Manual Character Consistency: The 3D model substitution step currently requires human assistance to select and place templates. The authors aim to automate this in the future.
  • Camera Angles: Professional storyboarding relies heavily on specific shot types (close-ups, wide shots, bird's-eye views). The current model doesn't explicitly reason about cinematic "picturing angles," which the authors identify as a key area for improvement.
  • Candidate Pool Dependency: The system's "inspiration" is only as good as its database. If the GraphMovie dataset doesn't contain a specific niche scene, the retriever may struggle.

Summary updated

Abstract

A storyboard is a sequence of images to illustrate a story containing multiple sentences, which has been a key process to create different story products. In this paper, we tackle a new multimedia task of automatic storyboard creation to facilitate this process and inspire human artists. Inspired by the fact that our understanding of languages is based on our past experience, we propose a novel inspire-and-create framework with a story-to-image retriever that selects relevant cinematic images for inspiration and a storyboard creator that further refines and renders images to improve the relevancy and visual consistency. The proposed retriever dynamically employs contextual information in the story with hierarchical attentions and applies dense visual-semantic matching to accurately retrieve and ground images. The creator then employs three rendering steps to increase the flexibility of retrieved images, which include erasing irrelevant regions, unifying styles of images and substituting consistent characters. We carry out extensive experiments on both in-domain and out-of-domain visual story datasets. The proposed model achieves better quantitative performance than the state-of-the-art baselines for storyboard creation. Qualitative visualizations and user studies further verify that our approach can create high-quality storyboards even for stories in the wild.
BibTeX
@inproceedings{chen2019neural,
  author = {Shizhe Chen and Bei Liu and Jianlong Fu and Ruihua Song and Qin Jin and Pingping Lin and Xiaoyu Qi and Chunting Wang and Jin Zhou},
  title = {{Neural Storyboard Artist: Visualizing Stories with Coherent Image Sequences}},
  booktitle = {ACM Multimedia},
  year = {2019},
  doi = {10.1145/3343031.3350571},
}

Loading related content...

Figure