Introduction
Make-A-Storyboard is a novel framework designed to solve the "consistency gap" in AI-generated storytelling. By disentangling the control of characters and scenes and merging them through a specialized "Balance-Aware" strategy, the system generates sequences of images where both the protagonist and the environment remain visually consistent across the entire narrative.
Problem Statement
While Text-to-Image (T2I) models like Stable Diffusion have become incredibly powerful, they struggle with Story Visualization—the ability to generate a coherent sequence of images for a narrative. Current state-of-the-art (SOTA) methods for image customization, such as DreamBooth or Custom Diffusion, excel at capturing specific characters but often fail to place them in consistent, specific scenes. Simultaneous fine-tuning on both characters and scenes typically leads to "feature entanglement," resulting in character identity loss or erratic background shifts between frames.
Proposed Method: Make-A-Storyboard
The authors propose a three-stage pipeline to achieve high-fidelity story visualization:
- Contextual Prompt Processing: Utilizing a Large Language Model (LLM) as a "director," the framework parses a story into discrete scenes. The LLM identifies characters and locations, rewriting prompts into character-focused "narrative" components and scene-focused "adverbial" components.
- Disentangled Concept Control: Character and scene concepts are learned independently. This separation prevents visual features from bleeding between the subject and its environment.
- Balance-Aware Merge & Alternative Control: This core innovation combines character and scene features using a spatial mask at a critical "middle" timestep during denoising. Merging too early loses identity, while merging too late creates a disjointed appearance. During final steps, the model alternates between concept-specific U-Nets to ensure balanced representation.
Evaluation and Results
The framework was evaluated using eight custom datasets and a benchmark of 2,400 images across five story themes. Make-A-Storyboard demonstrated quantitative superiority, outperforming baselines in both Text-alignment (0.752) and Image-alignment (0.738).
Qualitatively, the model maintains "film-like" consistency across multiple frames. Unlike baseline models that often lose character features or alter room layouts, Make-A-Storyboard preserves the identity of both the protagonist and the setting throughout the narrative sequence.
Key Insights
- The "Goldilocks" Timestep: The success of feature merging depends on timing. A "balance" point (around timestep 120 in a 200-step process) allows for structural identity while harmonizing the character with the scene's lighting and texture.
- Robustness through Disentanglement: Independent fine-tuning of concepts until the inference stage proves more stable than simultaneous training for complex multi-concept generation.
- LLM-Driven Structure: Offloading narrative logic to LLMs allows the diffusion model to focus exclusively on high-quality visual synthesis.
Limitations and Future Work
The model can struggle with spatial layouts when scenes contain an excessive number of characters. Future research may explore incorporating explicit positional control mechanisms, such as bounding box constraints, to better manage complex narrative actions and crowded scenes.
Summary updated