StoryAgent: Customized Storytelling Video Generation via Multi-Agent Collaboration

Panwen Hu, Jin Jiang, Jianqi Chen, Mingfei Han, Shengcai Liao, Xiaojun Chang, Xiaodan LiangSubmission to ICLR 2025 · Rejected

TL;DR

StoryAgent introduces a collaborative multi-agent framework tailored for Customized Storytelling Video Generation (CSVG)—synthesizing cohesive, multi-shot narrative videos centered around a user-specified character. By decomposing production into distinct roles (story writing, storyboard painting, and video animation) and introducing LoRA-BE alongside a generate-remove-redraw storyboard pipeline, StoryAgent maintains character identity both across different shots and within dynamic video motion.

What Problem Does This Paper Address?

While modern text-to-video (T2V) and image-to-video (I2V) models generate impressive standalone clips, producing a coherent, multi-shot narrative featuring an explicit, user-provided protagonist remains remarkably difficult. Existing pipelines suffer from two major failure modes:

  1. Inter-shot inconsistency: Characters change clothes, facial features, or art styles when transitioning between scenes.
  2. Intra-shot degradation: Subject identity warps, drifts, or dissolves during camera movements and complex character motions within a single shot.

Existing multi-agent systems (like Mora and AesopAgent) handle generic story-to-video generation but lack mechanisms to enforce a designated subject. Meanwhile, concept-personalization approaches (such as DreamVideo or Magic-Me) struggle to generalize out-of-domain identities across complex, narrative-driven scenes.

Methods

StoryAgent mirrors a real-world animation studio by assigning subtasks to specialized agents:

  • Agent Manager: An LLM orchestrator that schedules tasks, monitors pipeline state, and routes artifacts between agents.
  • Story Designer: Prompts an LLM (e.g., GPT-4) to expand user ideas into structured, $N$-shot cinematic scripts including background scenes, character actions, shot framing, and camera motions.
  • Storyboard Generator (Inter-shot consistency): Rather than asking a diffusion model to render the custom character from scratch, it executes a three-step Generation $\rightarrow$ Removal $\rightarrow$ Redrawing pipeline:
  • Generation: StoryDiffusion produces scene layouts based on shot descriptions.
  • Removal: LangSAM segments and masks out the generic character.
  • Redrawing: StoryAnyDoor (fine-tuned on the reference character's images/video frames) composites the exact customized subject into the masked region.
  • Video Creator with LoRA-BE (Intra-shot consistency): Built on the DynamiCrafter I2V backbone, it introduces LoRA with Block-wise Embeddings (LoRA-BE). To avoid character distortion:
  • LoRA is inserted into self-attention, cross-attention, and temporal attention blocks.
  • Rather than a single shared Textual Inversion token, 16 distinct token embeddings are trained—one injected into each of the 16 cross-attention blocks of the 3D U-Net.
  • A cross-attention localization loss ($L_{loc}$) forces the custom token's attention maps to strictly align with the ground-truth subject mask.
  • Observer (Optional): Evaluates visual quality via aesthetic predictors to flag frames for regeneration.
[User Prompt & Reference Video]
              │
      [Agent Manager]
              │
      [Story Designer]  ──> Script & Shot Prompts
              │
  [Storyboard Generator] ──> Generate Scene → Segment Mask → Redraw Custom Subject
              │
   [Video Creator (LoRA-BE)] ──> DynamiCrafter + Block-wise Embeddings + Temporal LoRA
              │
       [Final Story Video]

Datasets

  • PororoSV: Benchmark storytelling dataset; 5 cartoon characters were used as customized subjects, with 10 testing samples per character (4 shots each).
  • FlintstonesSV: Benchmark cartoon storytelling dataset; 4 characters were evaluated under the same 4-shot configuration.
  • Open-Domain Test Set: 8 new out-of-domain subjects collected by the authors from YouTube and open-source websites (e.g., Miffy, Hello Kitty, a cartoon elephant, and a realistic bird) paired with ChatGPT-generated narrative prompts.

Key Insights and Ideas

  • Decoupling layout generation from character insertion: Asking a text-to-image model to simultaneously solve multi-shot composition and novel character personalization leads to feature bleeding and hallucinations. Inpainting the personalized subject into a pre-composed background guarantees fine detail retention (e.g., exact clothing patterns).
  • Block-wise cross-attention embeddings: Standard Textual Inversion injects a single global embedding across all layers, which struggles to balance coarse structure against fine texture. Dedicating independent token vectors per U-Net block grants the network capacity to encode distinct multi-scale semantics.
  • MLLMs fail as automated aesthetic judges: In auxiliary experiments evaluating GPT-4o and Gemini as automated quality evaluators, the authors found that current multimodal LLMs frequently scored artifact-heavy generations higher than ground-truth animations. Simple aesthetic reward models proved slightly more reliable, but human-in-the-loop oversight remains necessary.

Key Results

  • Public Benchmarks: On FlintstonesSV, StoryAgent achieved an FVD of 991.37 (compared to SVD's 1864.91 and TI-SparseCtrl's 3277.96), while improving SSIM to 0.6700 (vs. 0.4460 for SVD) and reducing LPIPS to 0.2490. Similar margins were recorded on PororoSV (FVD 2070.56 vs. SVD's 2634.01).
  • Open-Domain Video Personalization: On the open-domain set, StoryAgent achieved the highest text-video alignment (CLIP-score 0.2053) while maintaining visual realism (IS 2.6346).
  • User Studies: In blind human evaluations across 5 metrics (1–5 scale), StoryAgent dominated baseline methods, notably earning 4.6 on Inter-shot Subject Consistency (vs. 3.4 for SVD) and 4.8 on Intra-shot Subject Consistency (vs. 3.0 for SVD).
  • Ablation on LoRA-BE: On FlintstonesSV, animating with naive DynamiCrafter fine-tuning yielded an FVD of 3753.91 and SSIM of 0.3357; adding LoRA-BE improved these metrics drastically to 991.37 and 0.6700, respectively.

Limitations and Future Work

  • Human Video Generation: The pipeline struggles with realistic, customized human subjects due to the underlying priors of open-source video models.
  • Short Shot Lengths: Each shot is constrained to short durations (roughly 1–2 seconds / 16 frames).
  • Single-Subject Focus: The current implementation personalizes one protagonist at a time; orchestrating multi-shot narratives featuring multiple interacting, personalized characters remains an open challenge.

Summary updated

Abstract

The advent of AI-Generated Content (AIGC) has spurred research into automated video generation to streamline conventional processes. However, automating storytelling video production, particularly for customized narratives, remains challenging due to the complexity of maintaining subject consistency across shots. While existing approaches like Mora and AesopAgent integrate multiple agents for Story-to-Video (S2V) generation, they fall short in preserving protagonist consistency and supporting Customized Storytelling Video Generation (CSVG). To address these limitations, we propose StoryAgent, a multi-agent framework designed for CSVG. StoryAgent decomposes CSVG into distinct subtasks assigned to specialized agents, mirroring the professional production process. Notably, our framework includes agents for story design, storyboard generation, video creation, agent coordination, and result evaluation. Leveraging the strengths of different models, StoryAgent enhances control over the generation process, significantly improving character consistency. Specifically, we introduce a customized Image-to-Video (I2V) method, LoRA-BE, to enhance intra-shot temporal consistency, while a novel storyboard generation pipeline is proposed to maintain subject consistency across shots. Extensive experiments demonstrate the effectiveness of our approach in synthesizing highly consistent storytelling videos, outperforming state-of-the-art methods. Our contributions include the introduction of StoryAgent, a versatile framework for video generation tasks, and novel techniques for preserving protagonist consistency.
BibTeX
@misc{hu2024storyagent,
  author = {Panwen Hu and Jin Jiang and Jianqi Chen and Mingfei Han and Shengcai Liao and Xiaojun Chang and Xiaodan Liang},
  title = {{StoryAgent: Customized Storytelling Video Generation via Multi-Agent Collaboration}},
  year = {2024},
  eprint = {2411.04925},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.04925},
}

Loading related content...

Figure