TL;DR
StoryAgent introduces a collaborative multi-agent framework tailored for Customized Storytelling Video Generation (CSVG)—synthesizing cohesive, multi-shot narrative videos centered around a user-specified character. By decomposing production into distinct roles (story writing, storyboard painting, and video animation) and introducing LoRA-BE alongside a generate-remove-redraw storyboard pipeline, StoryAgent maintains character identity both across different shots and within dynamic video motion.
What Problem Does This Paper Address?
While modern text-to-video (T2V) and image-to-video (I2V) models generate impressive standalone clips, producing a coherent, multi-shot narrative featuring an explicit, user-provided protagonist remains remarkably difficult. Existing pipelines suffer from two major failure modes:
- Inter-shot inconsistency: Characters change clothes, facial features, or art styles when transitioning between scenes.
- Intra-shot degradation: Subject identity warps, drifts, or dissolves during camera movements and complex character motions within a single shot.
Existing multi-agent systems (like Mora and AesopAgent) handle generic story-to-video generation but lack mechanisms to enforce a designated subject. Meanwhile, concept-personalization approaches (such as DreamVideo or Magic-Me) struggle to generalize out-of-domain identities across complex, narrative-driven scenes.
Methods
StoryAgent mirrors a real-world animation studio by assigning subtasks to specialized agents:
- Agent Manager: An LLM orchestrator that schedules tasks, monitors pipeline state, and routes artifacts between agents.
- Story Designer: Prompts an LLM (e.g., GPT-4) to expand user ideas into structured, $N$-shot cinematic scripts including background scenes, character actions, shot framing, and camera motions.
- Storyboard Generator (Inter-shot consistency): Rather than asking a diffusion model to render the custom character from scratch, it executes a three-step Generation $\rightarrow$ Removal $\rightarrow$ Redrawing pipeline:
- Generation: StoryDiffusion produces scene layouts based on shot descriptions.
- Removal: LangSAM segments and masks out the generic character.
- Redrawing: StoryAnyDoor (fine-tuned on the reference character's images/video frames) composites the exact customized subject into the masked region.
- Video Creator with LoRA-BE (Intra-shot consistency): Built on the DynamiCrafter I2V backbone, it introduces LoRA with Block-wise Embeddings (LoRA-BE). To avoid character distortion:
- LoRA is inserted into self-attention, cross-attention, and temporal attention blocks.
- Rather than a single shared Textual Inversion token, 16 distinct token embeddings are trained—one injected into each of the 16 cross-attention blocks of the 3D U-Net.
- A cross-attention localization loss ($L_{loc}$) forces the custom token's attention maps to strictly align with the ground-truth subject mask.
- Observer (Optional): Evaluates visual quality via aesthetic predictors to flag frames for regeneration.
[User Prompt & Reference Video]
│
[Agent Manager]
│
[Story Designer] ──> Script & Shot Prompts
│
[Storyboard Generator] ──> Generate Scene → Segment Mask → Redraw Custom Subject
│
[Video Creator (LoRA-BE)] ──> DynamiCrafter + Block-wise Embeddings + Temporal LoRA
│
[Final Story Video]
Datasets
- PororoSV: Benchmark storytelling dataset; 5 cartoon characters were used as customized subjects, with 10 testing samples per character (4 shots each).
- FlintstonesSV: Benchmark cartoon storytelling dataset; 4 characters were evaluated under the same 4-shot configuration.
- Open-Domain Test Set: 8 new out-of-domain subjects collected by the authors from YouTube and open-source websites (e.g., Miffy, Hello Kitty, a cartoon elephant, and a realistic bird) paired with ChatGPT-generated narrative prompts.
Key Insights and Ideas
- Decoupling layout generation from character insertion: Asking a text-to-image model to simultaneously solve multi-shot composition and novel character personalization leads to feature bleeding and hallucinations. Inpainting the personalized subject into a pre-composed background guarantees fine detail retention (e.g., exact clothing patterns).
- Block-wise cross-attention embeddings: Standard Textual Inversion injects a single global embedding across all layers, which struggles to balance coarse structure against fine texture. Dedicating independent token vectors per U-Net block grants the network capacity to encode distinct multi-scale semantics.
- MLLMs fail as automated aesthetic judges: In auxiliary experiments evaluating GPT-4o and Gemini as automated quality evaluators, the authors found that current multimodal LLMs frequently scored artifact-heavy generations higher than ground-truth animations. Simple aesthetic reward models proved slightly more reliable, but human-in-the-loop oversight remains necessary.
Key Results
- Public Benchmarks: On FlintstonesSV, StoryAgent achieved an FVD of 991.37 (compared to SVD's 1864.91 and TI-SparseCtrl's 3277.96), while improving SSIM to 0.6700 (vs. 0.4460 for SVD) and reducing LPIPS to 0.2490. Similar margins were recorded on PororoSV (FVD 2070.56 vs. SVD's 2634.01).
- Open-Domain Video Personalization: On the open-domain set, StoryAgent achieved the highest text-video alignment (CLIP-score 0.2053) while maintaining visual realism (IS 2.6346).
- User Studies: In blind human evaluations across 5 metrics (1–5 scale), StoryAgent dominated baseline methods, notably earning 4.6 on Inter-shot Subject Consistency (vs. 3.4 for SVD) and 4.8 on Intra-shot Subject Consistency (vs. 3.0 for SVD).
- Ablation on LoRA-BE: On FlintstonesSV, animating with naive DynamiCrafter fine-tuning yielded an FVD of 3753.91 and SSIM of 0.3357; adding LoRA-BE improved these metrics drastically to 991.37 and 0.6700, respectively.
Limitations and Future Work
- Human Video Generation: The pipeline struggles with realistic, customized human subjects due to the underlying priors of open-source video models.
- Short Shot Lengths: Each shot is constrained to short durations (roughly 1–2 seconds / 16 frames).
- Single-Subject Focus: The current implementation personalizes one protagonist at a time; orchestrating multi-shot narratives featuring multiple interacting, personalized characters remains an open challenge.
Summary updated