Abstract
A storyboard is a sequence of images that illustrates a multi-sentence story, serving as a critical step in creative production. We introduce the Neural Storyboard Artist, an automatic framework designed to facilitate this process. Our "Inspire-and-Create" approach utilizes a story-to-image retriever to select relevant cinematic images for inspiration, followed by a storyboard creator that refines these images for better relevancy and visual consistency. The retriever uses contextual information and dense visual-semantic matching, while the creator employs region segmentation, style unification, and character substitution. Experiments on VIST and a new GraphMovie dataset demonstrate that our model produces high-quality, coherent storyboards that outperform existing baselines.
Introduction
Creating a storyboard is a labor-intensive process that requires both artistic skill and narrative intuition. While generative models like GANs offer flexibility, they often struggle with the complexity of "in-the-wild" scenes, resulting in blurry or distorted images. Conversely, traditional retrieval-based methods provide high-quality cinematic images but often fail to maintain narrative coherence or visual consistency across a sequence. The Neural Storyboard Artist addresses these challenges by bridging the gap between high-quality retrieval and flexible rendering.
The Inspire-and-Create Framework
The proposed framework consists of two primary stages: the "Inspire" stage for context-aware image retrieval and the "Create" stage for visual refinement.
1. The "Inspire" Stage: Contextual-Aware Dense Matching
The Contextual-Aware Dense Matching (CADM) retriever moves beyond isolated sentence matching. It employs a hierarchical attention mechanism to incorporate surrounding story context, which is crucial for resolving pronouns and maintaining thematic continuity.
- Dense Matching: The system grounds specific words to corresponding image regions, ensuring that the visual content accurately reflects the textual details.
- One-to-Many Coverage: A greedy decoding algorithm allows for the retrieval of multiple complementary images for complex sentences, ensuring comprehensive coverage of the narrative.
2. The "Create" Stage: Storyboard Creator
After retrieval, the system applies three rendering steps to unify the visual sequence:
- Relevant Region Segmentation: By fusing dense matching grounding with Mask R-CNN, the system isolates story-relevant objects and removes distracting backgrounds.
- Style Unification: To ensure a consistent aesthetic, CartoonGAN is used to apply a uniform artistic style across the entire sequence.
- 3D Model Substitution: To maintain character consistency, 3D character templates are overlaid on retrieved images, ensuring the protagonist's appearance remains stable throughout the story.
Datasets and Evaluation
The model was pre-trained on MSCOCO and evaluated on the Visual Storytelling (VIST) dataset. Additionally, the authors introduced GraphMovie, a large-scale dataset of 3 million sentence-image pairs crawled from movie explanatory sites, providing a rich pool of cinematic candidates.
On the VIST dataset, the CADM model achieved a Recall@100 of 45.53%, a significant improvement over the state-of-the-art CNSI baseline (4.5%). Human evaluations further supported these findings, with CADM storyboards rated as "good" 52.2% of the time, compared to 38.4% for non-contextual models.
Conclusion and Future Work
The Neural Storyboard Artist demonstrates that leveraging inter-sentence context and high-quality retrieval foundations can produce professional-grade storyboards. Future work will focus on automating the character substitution process and explicitly modeling cinematic camera angles to further enhance the "director's vision" in automated storyboarding.
Summary updated