VideoGen-of-Thought: A Collaborative Framework for Multi-Shot Video Generation

Mingzhe Zheng, Yongqi Xu, Haojian Huang, Xuran Ma, Yexin Liu, Wenjie Shu, Yatian Pang, Feilong Tang, Qifeng Chen, Harry Yang, Ser-Nam Lim

Abstract

Current video generation models excel at short clips but fail to produce cohesive multi-shot narratives due to disjointed visual dynamics and fractured storylines. Existing solutions either rely on extensive manual scripting/editing or prioritize single-shot fidelity over cross-scene continuity, limiting their practicality for movie-like content. We introduce VideoGen-of-Thought (VGoT), a step-by-step framework that automates multi-shot video synthesis from a single sentence by systematically addressing three core challenges: (1) Narrative Fragmentation: Existing methods lack structured storytelling. We propose dynamic storyline modeling, which first converts the user prompt into concise shot descriptions, then elaborates them into detailed, cinematic specifications across five domains (character dynamics, background continuity, relationship evolution, camera movements, HDR lighting), ensuring logical narrative progression with self-validation. (2) Visual Inconsistency: Existing approaches struggle with maintaining visual consistency across shots. Our identity-aware cross-shot propagation generates identity-preserving portrait (IPP) tokens that maintain character fidelity while allowing trait variations (expressions, aging) dictated by the storyline. (3) Transition Artifacts: Abrupt shot changes disrupt immersion. Our adjacent latent transition mechanisms implement boundary-aware reset strategies that process adjacent shots' features at transition points, enabling seamless visual flow while preserving narrative continuity. VGoT generates multi-shot videos that outperform state-of-the-art baselines by 20.4% in within-shot face consistency and 17.4% in style consistency, while achieving over 100% better cross-shot consistency and 10x fewer manual adjustments than alternatives.

Introduction

Generating a stunning five-second clip of a cat in space is now a solved problem. However, generating a coherent, multi-shot short film that tells a story remains a significant challenge characterized by fractured plots, characters who change appearance between cuts, and jarring transitions.

Researchers from HKUST, Peking University, and Everlyn AI have introduced VideoGen-of-Thought (VGoT), a framework that moves beyond simple "prompt-to-video" into the realm of "sentence-to-cinema." By applying a step-by-step reasoning approach—reminiscent of Chain-of-Thought in Large Language Models (LLMs)—to video synthesis, VGoT produces consistent, multi-shot narratives with 10x less manual intervention than previous state-of-the-art methods.

Addressing the Core Challenges

Current video generation models (like Sora or Kling) excel at single, short clips but struggle with multi-shot consistency. The authors identify three core "villains" in multi-shot generation that VGoT is designed to defeat:

  1. Narrative Fragmentation: Storylines often lack logic or cinematic structure when generated shot-by-shot without a global plan.
  2. Visual Inconsistency: Characters and styles shift wildly between cuts, leading to "identity drift."
  3. Transition Artifacts: Abrupt jumps between shots break the viewer's immersion.
Comparison of VGoT against state-of-the-art baselines and an overview of the three core challenges in multi-shot video generation.

The VGoT Framework

VGoT operates through a structured, four-module pipeline that orchestrates existing tools (like GPT-4o for scripting and DynamiCrafter for synthesis) into a cohesive system without requiring additional training.

Dynamic Storyline Modeling (DSM)

Instead of feeding a single prompt to a video model, VGoT uses an LLM to expand a sentence into a series of "shot drafts." These are refined into detailed specifications across five cinematic domains: character dynamics, background, relationships, camera movement, and HDR lighting. A self-validation loop ensures these scripts adhere to cinematic principles and narrative logic.

Identity-Aware Cross-Shot Propagation

To maintain character fidelity, the system extracts Identity-Preserving Portrait (IPP) tokens. These act as a visual anchor for a character's facial structure while allowing for "intra-story evolution," such as aging or changing expressions, as dictated by the narrative.

Adjacent Latent Transition Mechanisms

To eliminate "jump cuts," VGoT manages noise at shot boundaries. A boundary-aware noise reset strategy processes the features of adjacent shots together, ensuring a seamless visual flow from the end of one shot to the beginning of the next.

The VGoT pipeline, illustrating the flow from user prompt to shot descriptions, identity-preserving portrait tokens, and final shot-by-shot transitions.

Experimental Evaluation

Because existing video datasets are mostly single-shot clips, the authors constructed a new Multi-Shot Benchmark consisting of ten 30-shot stories. Scenarios range from longitudinal character development to complex multi-actor scenes.

VGoT significantly outperformed baselines like VideoCrafter2 and EasyAnimate across several key metrics:

  • Face Consistency: Improved within-shot face consistency by 20.4% and cross-shot consistency by 2.9x.
  • Style Consistency: Boosted style uniformity across shots by 17.4%.
  • Human Preference: In user studies, VGoT received 66.7% "Good" ratings for cross-shot consistency, compared to just 27.2% for its closest competitor.

Key Insights and Limitations

A notable finding was the "CLIP-Logic Tradeoff": VGoT sometimes resulted in lower CLIP scores than simpler baselines. This occurred because baselines produce generic images that match simple prompts perfectly, whereas VGoT produces complex, narratively-driven scenes that prioritize cinematic logic over literal prompt-matching.

While VGoT is a massive step forward, it remains limited by the motion diversity of its underlying video backbones. Maintaining consistency between non-adjacent shots (e.g., Shot 1 and Shot 30) also remains a challenge. Future work will focus on integrating stronger video backbones and handling multi-subject identity disentanglement.

Summary updated

Abstract

Current video generation models excel at short clips but fail to produce cohesive multi-shot narratives due to disjointed visual dynamics and fractured storylines. Existing solutions either rely on extensive manual scripting/editing or prioritize single-shot fidelity over cross-scene continuity, limiting their practicality for movie-like content. We introduce VideoGen-of-Thought (VGoT), a step-by-step framework that automates multi-shot video synthesis from a single sentence by systematically addressing three core challenges: (1) Narrative Fragmentation: Existing methods lack structured storytelling. We propose dynamic storyline modeling, which first converts the user prompt into concise shot descriptions, then elaborates them into detailed, cinematic specifications across five domains (character dynamics, background continuity, relationship evolution, camera movements, HDR lighting), ensuring logical narrative progression with self-validation. (2) Visual Inconsistency: Existing approaches struggle with maintaining visual consistency across shots. Our identity-aware cross-shot propagation generates identity-preserving portrait (IPP) tokens that maintain character fidelity while allowing trait variations (expressions, aging) dictated by the storyline. (3) Transition Artifacts: Abrupt shot changes disrupt immersion. Our adjacent latent transition mechanisms implement boundary-aware reset strategies that process adjacent shots' features at transition points, enabling seamless visual flow while preserving narrative continuity. VGoT generates multi-shot videos that outperform state-of-the-art baselines by 20.4% in within-shot face consistency and 17.4% in style consistency, while achieving over 100% better cross-shot consistency and 10x fewer manual adjustments than alternatives.
BibTeX
@misc{zheng2025videogen,
  author = {Mingzhe Zheng and Yongqi Xu and Haojian Huang and Xuran Ma and Yexin Liu and Wenjie Shu and Yatian Pang and Feilong Tang and Qifeng Chen and Harry Yang and Ser-Nam Lim},
  title = {{VideoGen-of-Thought: A Collaborative Framework for Multi-Shot Video Generation}},
  year = {2025},
  eprint = {2503.15138},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.15138},
}
Figure