TL;DR
Researchers have developed the Neural Storyboard Artist, a framework that transforms multi-sentence stories into coherent visual sequences. By combining a context-aware image retriever with a rendering pipeline that unifies styles and characters, the system moves beyond simple text-to-image matching to create "cinematic" storyboards that maintain narrative flow.
What Problem Does This Paper Address?
Creating a storyboard—a sequence of images used to plan movies or animations—is a labor-intensive task requiring artistic skill and narrative intuition. Automating this is difficult for two main reasons:
- The Quality-Flexibility Trade-off: Generative models (like GANs) are flexible but often produce blurry or distorted images for complex, "in-the-wild" scenes. Retrieval-based methods provide high-quality "cinematic" images but are often rigid, failing to account for story context or visual consistency across the sequence.
- Narrative Coherence: A storyboard isn't just a collection of random pictures; it needs a consistent style and the same characters appearing across different shots. Most existing models treat sentences in isolation, losing the "thread" of the story.
Methods
The authors propose an "Inspire-and-Create" framework consisting of two main stages:
1. The "Inspire" Stage: Contextual-Aware Dense Matching (CADM)
Instead of looking at one sentence at a time, the CADM retriever uses a hierarchical attention mechanism. It looks at the current sentence and the surrounding story context to understand pronouns (e.g., knowing "they" refers to "mom and daughter" mentioned earlier).
- Dense Matching: It doesn't just match a sentence to an image; it grounds specific words to specific regions in an image.
- One-to-Many Coverage: A greedy decoding algorithm allows the system to retrieve multiple complementary images for a single long or complex sentence, ensuring no story detail is left out.
2. The "Create" Stage: Storyboard Creator
Once images are retrieved, the system "simulates" an artist's touch through three rendering steps:
- Relevant Region Segmentation: To remove distracting backgrounds, the system fuses the retriever’s "word-to-region" grounding with Mask R-CNN to precisely isolate and keep only the objects/scenes relevant to the story.
- Style Unification: To prevent the storyboard from looking like a disjointed collage, it uses CartoonGAN to apply a consistent artistic style across all images.
- 3D Model Substitution: To solve the "character consistency" problem, the authors use a semi-manual step where 3D character templates (via Autodesk Maya) are placed over the retrieved images to ensure the protagonist looks the same in every frame.
Datasets
- MSCOCO: Used for pre-training the basic image-text matching.
- VIST (Visual Storytelling): The primary benchmark for sequential image-to-story tasks.
- GraphMovie (New): A large-scale dataset crawled from movie explanatory websites, containing over 3 million aligned sentence-image pairs. This provides the "cinematic" candidate pool for the retriever.
Key Insights and Ideas
- Context is King: The paper demonstrates that inter-sentence context significantly boosts retrieval accuracy. Understanding the "flow" of the story helps the model pick images that make sense as a sequence, not just as individual frames.
- Retrieval as a Foundation: Rather than trying to generate pixels from scratch (which is hard for complex scenes), the authors argue that retrieving a high-quality cinematic base and then modifying it is a more viable path for professional-grade storyboarding.
- Complementary Segmentation: A clever insight was fusing "Dense Matching" (which knows what is relevant but has messy boundaries) with "Mask R-CNN" (which has perfect boundaries but doesn't know what is relevant to the story).
Key Results
- Quantitative: On the VIST dataset, the CADM model achieved a Recall@100 of 45.53%, significantly outperforming the state-of-the-art baseline (CNSI) which sat at a mere 4.5%.
- Qualitative: In human evaluations, the CADM-generated storyboards were rated as "good" 52.2% of the time, compared to 38.4% for models that didn't use story context.
- Generalization: The model proved robust even for "stories in the wild," such as Chinese idioms and movie scripts that weren't in the training set.
Limitations and Future Work
- Semi-Manual Character Consistency: The 3D model substitution step currently requires human assistance to select and place templates. The authors aim to automate this in the future.
- Camera Angles: Professional storyboarding relies heavily on specific shot types (close-ups, wide shots, bird's-eye views). The current model doesn't explicitly reason about cinematic "picturing angles," which the authors identify as a key area for improvement.
- Candidate Pool Dependency: The system's "inspiration" is only as good as its database. If the GraphMovie dataset doesn't contain a specific niche scene, the retriever may struggle.
Summary updated