VideoGen-of-Thought: Step-by-step generating multi-shot video with minimal manual intervention

Mingzhe Zheng, Yongqi Xu, Haojian Huang, Xuran Ma, Yexin Liu, Wenjie Shu, Yatian Pang, Feilong Tang, Qifeng Chen, Harry Yang, Ser-Nam Lim

TL;DR

This paper introduces VideoGen-of-Thought (VGoT), a novel framework for generating coherent, multi-shot videos from a single sentence prompt with minimal human intervention. VGoT tackles the long-standing challenges of narrative fragmentation, visual inconsistency, and jarring transitions, enabling director-level storytelling for automated video creation.

What Problem Does This Paper Address?

Current text-to-video models are adept at generating short, visually appealing clips but struggle significantly when it comes to creating longer, multi-shot narratives. The core issues are:

  • Fractured Storylines: Existing methods often produce disjointed narratives that lack logical progression and depth across scenes.
  • Visual Inconsistency: Maintaining consistent character appearance, background, and overall style across multiple shots is a major hurdle.
  • Abrupt Transitions: Shot changes can be jarring, breaking the immersion and flow of the video.

Existing solutions often require extensive manual scripting, editing, or prioritize single-shot quality over cross-scene continuity, making them impractical for automated, movie-like content generation. VGoT aims to bridge this gap by providing a systematic, automated approach to multi-shot video synthesis.

Methods

VGoT employs a structured, step-by-step framework to address the core challenges:

  1. Dynamic Storyline Modeling with Self-Validation:
    • LLM-Powered Decomposition: A user's single-sentence prompt is first broken down into concise shot descriptions.
  2. Elaboration and Cinematic Specifications: These descriptions are then elaborated into detailed cinematic specifications across five domains: character dynamics, background continuity, relationship evolution, camera movements, and HDR lighting.
  3. Self-Validation: A mechanism ensures narrative coherence by checking for logical progression and adherence to cinematic principles, rejecting candidates that violate them. This is achieved through a two-stage process involving an LLM and rule-based checks.
  4. Identity-Aware Cross-Shot Propagation:
    • Identity-Preserving Portrait (IPP) Tokens: To combat visual inconsistency, VGoT generates IPP tokens. These tokens capture character identity while allowing for story-driven trait variations (e.g., expressions, aging).
  5. Hierarchical Feature Injection: These IPP tokens are then injected into pre-trained diffusion models via cross-attention to guide keyframe generation, ensuring consistent character representation and style across shots.
  6. Adjacent Latent Transition Mechanisms:
    • Boundary-Aware Reset: To smooth transitions between shots, VGoT implements a "boundary-aware reset" strategy. This mechanism processes features from adjacent shots at transition points, enabling a seamless visual flow while preserving narrative continuity. This is inspired by FIFO (First-In, First-Out) principles.

Datasets

The paper mentions constructing a benchmark dataset for evaluation:

  • Custom Benchmark Dataset: VGoT was used to create ten 30-shot stories (300 shots in total) for evaluation. Each story originates from a single user input, generating shot outlines and detailed scripts.

The paper also references existing datasets used for training the underlying diffusion models, such as those for large-scale image-text pairs (e.g., LAION).

Key Insights and Ideas

  • Decomposition is Key: Breaking down the complex problem of multi-shot video generation into distinct, manageable challenges (narrative, visual consistency, transitions) is crucial for success.
  • Storyline as the Foundation: A well-defined, dynamically generated storyline is paramount for guiding both visual consistency and narrative coherence across shots.
  • Identity Preservation Beyond Static Appearance: The concept of "identity-preserving portrait" tokens is innovative, allowing for character evolution (aging, expression changes) while maintaining core identity fidelity.
  • Transition Smoothing is More Than Just Blending: The adjacent latent transition mechanism highlights the importance of considering shot boundaries and their latent representations for seamless flow.
  • Balancing Consistency and Diversity: The ablation studies reveal a trade-off between narrative diversity and strict consistency, suggesting that a balanced approach is necessary for compelling video generation.

Key Results

  • Quantitative Improvements: VGoT achieves significant improvements over state-of-the-art baselines:
  • 20.4% higher within-shot face consistency (WS-FC).
  • 17.4% higher within-shot style consistency (WS-SC).
  • Over 100% improvement in cross-shot consistency metrics (CS-FC and CS-SC).
  • Reduced Manual Intervention: VGoT requires 10x fewer manual adjustments compared to alternatives like MovieDreamer and DreamFactory.
  • Human Evaluation: VGoT received 66.7% "Good" ratings for cross-shot consistency, significantly outperforming competitors (e.g., 27.2% for VideoCrafter2). In visual quality, 70.0% of evaluators rated VGoT outputs as "Good."
  • Ablation Study Insights: Removing the Dynamic Storyline Modeling (DSM) module led to narrative diversity loss, while the synergy between DSM and Identity-Preserving Portraits (IPP) was essential for optimal performance.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that current video datasets lack sufficient multi-shot narratives with consistent characters across scenes, necessitating the creation of a custom benchmark.
  • While VGoT significantly reduces manual intervention, some level of human oversight might still be beneficial for highly complex or nuanced narratives.
  • The paper mentions a "CLIP-Logic Tradeoff," where the full model achieves a lower CLIP score but higher PSNR and IS, suggesting a prioritization of cinematic quality over literal prompt matching.
  • Future Work:
    • The authors suggest that their work redefines automated long-video generation, bridging the gap between raw visual synthesis and director-level storytelling. This implies a direction towards more sophisticated narrative control and artistic expression in automated video generation.
  • The framework's modularity suggests potential for further enhancements in each component, such as more advanced storyline generation or finer-grained control over visual attributes.

Summary updated

Abstract

Current video generation models excel at short clips but fail to produce cohesive multi-shot narratives due to disjointed visual dynamics and fractured storylines. Existing solutions either rely on extensive manual scripting/editing or prioritize single-shot fidelity over cross-scene continuity, limiting their practicality for movie-like content. We introduce VideoGen-of-Thought (VGoT), a step-by-step framework that automates multi-shot video synthesis from a single sentence by systematically addressing three core challenges: (1) Narrative Fragmentation: Existing methods lack structured storytelling. We propose dynamic storyline modeling, which first converts the user prompt into concise shot descriptions, then elaborates them into detailed, cinematic specifications across five domains (character dynamics, background continuity, relationship evolution, camera movements, HDR lighting), ensuring logical narrative progression with self-validation. (2) Visual Inconsistency: Existing approaches struggle with maintaining visual consistency across shots. Our identity-aware cross-shot propagation generates identity-preserving portrait (IPP) tokens that maintain character fidelity while allowing trait variations (expressions, aging) dictated by the storyline. (3) Transition Artifacts: Abrupt shot changes disrupt immersion. Our adjacent latent transition mechanisms implement boundary-aware reset strategies that process adjacent shots' features at transition points, enabling seamless visual flow while preserving narrative continuity. VGoT generates multi-shot videos that outperform state-of-the-art baselines by 20.4% in within-shot face consistency and 17.4% in style consistency, while achieving over 100% better cross-shot consistency and 10x fewer manual adjustments than alternatives.
BibTeX
@misc{zheng2025videogen,
  author = {Mingzhe Zheng and Yongqi Xu and Haojian Huang and Xuran Ma and Yexin Liu and Wenjie Shu and Yatian Pang and Feilong Tang and Qifeng Chen and Harry Yang and Ser-Nam Lim},
  title = {{VideoGen-of-Thought: Step-by-step generating multi-shot video with minimal manual intervention}},
  year = {2025},
  eprint = {2503.15138},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.15138},
}
Figure