VideoGen-of-Thought: Step-by-step generating multi-shot video with minimal manual intervention

Mingzhe Zheng, Yang Xu, Haojian Huang, Xiaoliang Ma, Yexin Liu, Wei Shu, Yatian Pang, Feilong Tang, Qifeng Chen, Harry Yang, Ser-Nam Lim

Abstract

Current video generation models excel at short clips but fail to produce cohesive multi-shot narratives due to disjointed visual dynamics and fractured storylines. Existing solutions either rely on extensive manual scripting/editing or prioritize single-shot fidelity over cross-scene continuity, limiting their practicality for movie-like content. We introduce VideoGen-of-Thought (VGoT), a step-by-step framework that automates multi-shot video synthesis from a single sentence by systematically addressing three core challenges: (1) Narrative fragmentation: Existing methods lack structured storytelling. We propose dynamic storyline modeling, which turns the user prompt into concise shot drafts and then expands them into detailed specifications across five domains (character dynamics, background continuity, relationship evolution, camera movements, and HDR lighting) with self-validation to ensure logical progress. (2) Visual inconsistency: previous approaches struggle to maintain consistent appearance across shots. Our identity-aware cross-shot propagation builds identity-preserving portrait (IPP) tokens that keep character identity while allowing controlled trait changes (expressions, aging) required by the story. (3) Transition artifacts: Abrupt shot changes disrupt immersion. Our adjacent latent transition mechanisms implement boundary-aware reset strategies that process adjacent shots' features at transition points, enabling seamless visual flow while preserving narrative continuity. Combined in a training-free pipeline, VGoT surpasses strong baselines by 20.4% in within-shot face consistency and 17.4% in style consistency, while requiring 10x fewer manual adjustments. VGoT bridges the gap between raw visual synthesis and director-level storytelling for automated multi-shot video generation.

Introduction: The Multi-Shot Video Generation Challenge

While current AI models can generate stunning 2-second clips, they struggle to tell a cohesive story. Generating a "movie" using standard tools typically results in three major failures: narrative fragmentation, where characters appear and disappear without logic; visual inconsistency, where art styles or faces shift between shots; and transition artifacts, where jarring jumps break immersion.

VGoT addresses narrative fragmentation, visual inconsistency, and transition artifacts to produce more reasonable and consistent multi-shot videos compared to existing baselines.

Existing solutions often require "director-level" manual effort—writing detailed scripts for every shot and manually selecting reference images—making them impractical for automated content creation. VGoT aims to bridge this gap, moving from raw visual synthesis to automated storytelling by mimicking the professional film production process.

The VGoT Framework

VGoT operates as a structured, training-free pipeline that sits on top of existing diffusion models. Its architecture consists of three core modules designed to handle the complexities of multi-shot synthesis.

The VGoT pipeline: From a user prompt to detailed shot descriptions, identity-preserving portraits, and smooth shot-by-shot transitions.

Dynamic Storyline Modeling (DSM)

Using a Large Language Model (LLM), the system decomposes a single prompt into a sequence of shots. It expands these into detailed specifications across five cinematic domains: character dynamics, background continuity, relationship evolution, camera movement, and HDR lighting. A self-validation mechanism automatically rejects and regenerates shots that violate cinematic logic or narrative coherence, ensuring the story progresses logically.

Identity-Aware Cross-Shot Propagation

To maintain character consistency, VGoT generates Identity-Preserving Portrait (IPP) tokens. These tokens act as a visual anchor, guiding the diffusion process to maintain the character's core identity (facial structure, hair) while allowing for narrative-driven changes like aging or different expressions. This solves the "consistency-diversity paradox" where models often default to repetitive scenes to avoid identity shifts.

Adjacent Latent Transition Mechanisms

To smooth out the "jump cuts" between shots, VGoT implements a boundary-aware reset strategy. By processing the latent features of adjacent shots at their transition points, the model ensures a seamless visual flow, preventing the flicker or sudden shifts common in multi-clip generation.

Key Results and Insights

VGoT significantly outperforms existing models across both quantitative and qualitative metrics. By forcing the model to "think" through lighting, camera angles, and character arcs before pixels are drawn, the resulting output is far more coherent.

  • Superior Consistency: VGoT improved within-shot face consistency by 20.4% and style consistency by 17.4% over the best baselines. In cross-shot tasks, it achieved 2.9x higher face consistency than competitors.
  • Efficiency: The system requires 10x fewer manual adjustments to produce a coherent multi-shot sequence compared to previous state-of-the-art methods.
  • Human Preference: In a user study, VGoT received "Good" ratings for cross-shot consistency 66.7% of the time, compared to just 27.2% for the next-best baseline.

Conclusion

VGoT represents a shift toward automated director-level video generation. While it remains dependent on the quality of underlying base models, its training-free nature makes it highly adaptable. Future work will focus on improving long-range dependencies—such as characters returning to a scene after many shots—and expanding script generation to follow professional screenplay formats.

Summary updated

Abstract

Current video generation models excel at short clips but fail to produce cohesive multi-shot narratives due to disjointed visual dynamics and fractured storylines. Existing solutions either rely on extensive manual scripting/editing or prioritize single-shot fidelity over cross-scene continuity, limiting their practicality for movie-like content. We introduce VideoGen-of-Thought (VGoT), a step-by-step framework that automates multi-shot video synthesis from a single sentence by systematically addressing three core challenges: (1) Narrative fragmentation: Existing methods lack structured storytelling. We propose dynamic storyline modeling, which turns the user prompt into concise shot drafts and then expands them into detailed specifications across five domains (character dynamics, background continuity, relationship evolution, camera movements, and HDR lighting) with self-validation to ensure logical progress. (2) Visual inconsistency: previous approaches struggle to maintain consistent appearance across shots. Our identity-aware cross-shot propagation builds identity-preserving portrait (IPP) tokens that keep character identity while allowing controlled trait changes (expressions, aging) required by the story. (3) Transition artifacts: Abrupt shot changes disrupt immersion. Our adjacent latent transition mechanisms implement boundary-aware reset strategies that process adjacent shots' features at transition points, enabling seamless visual flow while preserving narrative continuity. Combined in a training-free pipeline, VGoT surpasses strong baselines by 20.4\% in within-shot face consistency and 17.4\% in style consistency, while requiring 10x fewer manual adjustments. VGoT bridges the gap between raw visual synthesis and director-level storytelling for automated multi-shot video generation.
BibTeX
@misc{zheng2025videogen,
  author = {Mingzhe Zheng and Yang Xu and Haojian Huang and Xiaoliang Ma and Yexin Liu and Wei Shu and Yatian Pang and Feilong Tang and Qifeng Chen and Harry Yang and Ser-Nam Lim},
  title = {{VideoGen-of-Thought: Step-by-step generating multi-shot video with minimal manual intervention}},
  year = {2025},
  eprint = {2503.15138},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.15138},
}
Figure