Make-A-Storyboard: A General Framework for Storyboard with Disentangled and Merged Control

Sitong Su, Litao Guo, Lianli Gao, Heng Tao Shen, Jingkuan SongarXiv 2023 · DOI · arXiv

Introduction

Make-A-Storyboard is a novel framework designed to solve the "consistency gap" in AI-generated storytelling. By disentangling the control of characters and scenes and merging them through a specialized "Balance-Aware" strategy, the system generates sequences of images where both the protagonist and the environment remain visually consistent across the entire narrative.

Problem Statement

While Text-to-Image (T2I) models like Stable Diffusion have become incredibly powerful, they struggle with Story Visualization—the ability to generate a coherent sequence of images for a narrative. Current state-of-the-art (SOTA) methods for image customization, such as DreamBooth or Custom Diffusion, excel at capturing specific characters but often fail to place them in consistent, specific scenes. Simultaneous fine-tuning on both characters and scenes typically leads to "feature entanglement," resulting in character identity loss or erratic background shifts between frames.

Proposed Method: Make-A-Storyboard

The authors propose a three-stage pipeline to achieve high-fidelity story visualization:

  1. Contextual Prompt Processing: Utilizing a Large Language Model (LLM) as a "director," the framework parses a story into discrete scenes. The LLM identifies characters and locations, rewriting prompts into character-focused "narrative" components and scene-focused "adverbial" components.
  2. Disentangled Concept Control: Character and scene concepts are learned independently. This separation prevents visual features from bleeding between the subject and its environment.
  3. Balance-Aware Merge & Alternative Control: This core innovation combines character and scene features using a spatial mask at a critical "middle" timestep during denoising. Merging too early loses identity, while merging too late creates a disjointed appearance. During final steps, the model alternates between concept-specific U-Nets to ensure balanced representation.
Reference concepts for character (Loopy), house, and sea used for disentangled control.

Evaluation and Results

The framework was evaluated using eight custom datasets and a benchmark of 2,400 images across five story themes. Make-A-Storyboard demonstrated quantitative superiority, outperforming baselines in both Text-alignment (0.752) and Image-alignment (0.738).

Qualitatively, the model maintains "film-like" consistency across multiple frames. Unlike baseline models that often lose character features or alter room layouts, Make-A-Storyboard preserves the identity of both the protagonist and the setting throughout the narrative sequence.

A generated storyboard sequence demonstrating visual consistency in character and setting across multiple narrative steps.

Key Insights

  • The "Goldilocks" Timestep: The success of feature merging depends on timing. A "balance" point (around timestep 120 in a 200-step process) allows for structural identity while harmonizing the character with the scene's lighting and texture.
  • Robustness through Disentanglement: Independent fine-tuning of concepts until the inference stage proves more stable than simultaneous training for complex multi-concept generation.
  • LLM-Driven Structure: Offloading narrative logic to LLMs allows the diffusion model to focus exclusively on high-quality visual synthesis.

Limitations and Future Work

The model can struggle with spatial layouts when scenes contain an excessive number of characters. Future research may explore incorporating explicit positional control mechanisms, such as bounding box constraints, to better manage complex narrative actions and crowded scenes.

Summary updated

Abstract

Story Visualization aims to generate images aligned with story prompts, reflecting the coherence of storybooks through visual consistency among characters and scenes.Whereas current approaches exclusively concentrate on characters and neglect the visual consistency among contextually correlated scenes, resulting in independent character images without inter-image coherence.To tackle this issue, we propose a new presentation form for Story Visualization called Storyboard, inspired by film-making, as illustrated in Fig.1.Specifically, a Storyboard unfolds a story into visual representations scene by scene. Within each scene in Storyboard, characters engage in activities at the same location, necessitating both visually consistent scenes and characters.For Storyboard, we design a general framework coined as Make-A-Storyboard that applies disentangled control over the consistency of contextual correlated characters and scenes and then merge them to form harmonized images.Extensive experiments demonstrate 1) Effectiveness.the effectiveness of the method in story alignment, character consistency, and scene correlation; 2) Generalization. Our method could be seamlessly integrated into mainstream Image Customization methods, empowering them with the capability of story visualization.
BibTeX
@misc{su2023make,
  author = {Sitong Su and Litao Guo and Lianli Gao and Heng Tao Shen and Jingkuan Song},
  title = {{Make-A-Storyboard: A General Framework for Storyboard with Disentangled and Merged Control}},
  year = {2023},
  eprint = {2312.07549},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.07549},
}
Figure