CaT-Movie: Balancing Global Planning and Local Rendering for Long Multi-Shot Video Generation

Jinpeng Lu, Haoyuan Shi, Yuanshen Guan, Jiyuan Xia, Bin Wu, Yinda Chen, Zhiwei XiongarXiv 2026 · arXiv

TL;DR

CaT-Movie is a framework for long-form, multi-shot video generation that maintains character and scene consistency across extended durations. By decoupling global narrative planning (via an LLM-driven storyboard) from local rendering (using identity-routing attention), it generates coherent movies that successfully transition between diverse camera setups without losing visual identity.

What Problem Does This Paper Address?

Generating a single high-quality video clip is now common, but stitching them into a coherent "movie" remains a massive challenge. Current multi-shot generators suffer from Coherence across Time (CaT) issues.

There are two main failure modes:

  1. Identity Drift: Characters or scenes gradually change their appearance as the video progresses across different generation windows.
  2. Copied Framing: To maintain consistency, some models "cheat" by repeating the exact same camera angle and background from previous shots, failing to produce the dynamic transitions (cuts) expected in cinema.

Existing benchmarks often reward this "cheating" because they measure consistency through simple feature similarity, which is highest when the frames are nearly identical.

Methods

CaT-Movie splits the task into two specialized components:

  • Global Story Planner: A training-free module that uses a Large Language Model (Qwen3.8-27B) to expand a user's prompt into a structured JSON storyboard. This storyboard tracks entities, locations, camera settings, and "state" (e.g., what a character is wearing or holding). Deterministic code then maintains this state across shots, ensuring that if a character picks up a hat in Shot 1, they are still holding it in Shot 5.
  • Local Window Renderer: A video diffusion transformer (fine-tuned from Wan2.2) that renders shots in small windows. It uses Reference-Routed Sparse Self-Attention to "borrow" identity features from specific "anchor views" (historical frames of the same character) indexed by the planner. This allows the model to maintain character likeness while the Shot-wise Cross-Attention ensures each shot follows its unique textual description and camera setup.
Overview of the CaT-Movie architecture showing the Planner and Renderer pipeline.

Datasets

The authors addressed the lack of high-quality multi-shot data by creating:

  • Film Storyboard Dataset: A massive collection of over 350,000 film shots annotated with entity registries, semantic state transitions, and grounding (boxes/masks).
  • CaT-MBench: A new benchmark specifically designed to test long-term consistency. Its key innovation is the "cut gate": a consistency check only counts as a "pass" if the model also successfully executed a planned camera setup change. This prevents models that simply "copy-paste" previous frames from scoring well.

Key Insights and Ideas

  • The "Cut Gate" Philosophy: True cinematic consistency isn't just about things looking the same; it's about things looking the same from a different perspective. By gating evaluation on successful cuts, the authors force the model to learn identity, not just pixels.
  • LLM as a "State Machine": Instead of asking the video model to remember everything, the LLM acts as a persistent memory of the "world state," which is then fed to the renderer as precise instructions.
  • Anchor Views over Recency: Most memory-augmented models look at the most recent frames. CaT-Movie looks for the best frames (clearest view, latest attire change) of a specific entity, regardless of how far back they occurred.
Qualitative comparison showing CaT-Movie maintaining character identity across different camera setups compared to baselines.

Key Results

  • Breaking the Pareto Frontier: As shown in Figure 4, CaT-Movie is the only method that achieves both high transition control (executing planned cuts) and high inter-shot consistency.
  • Quantitative Dominance: On CaT-MBench, CaT-Movie achieved the highest average score (0.597) and the most consistent characters (0.651) among all tested methods, including StoryMem and HoloCine.
  • Cut Execution: It successfully executed 97.4% of planned cuts, significantly outperforming other joint generators that often "merge" shots to maintain consistency.
The Pareto frontier of consistency vs. transition control, highlighting CaT-Movie's superior performance.

Limitations and Future Work

  • Instruction Following: While the model is excellent at persisting a state, it sometimes struggles to establish a new requested change (like a character suddenly putting on a hat) if the base video model's prior is too strong.
  • Computational Overhead: Running a 27B parameter LLM for planning and a 5B parameter transformer for rendering is computationally intensive.
  • Future Directions: The authors suggest that better integration of 3D-aware priors could further improve consistency when camera angles change drastically (e.g., a 180-degree reverse shot).

Summary updated

Abstract

Current multi-shot video generators render only limited durations per pass, requiring long movies to be stitched from local segments across which narrative progression and visual identity drift. To synthesize long-horizon movies with Coherence across Time, we present CaT-Movie, a multi-shot generation system consisting of a global story planner and a local window renderer. The planner expands user requests via a large language model to preserve, infer, and complete story details, while deterministic code maintains the resulting storyboard across shots to compile self-contained shot prompts and index reference frames from generated history. To execute this guidance, the renderer applies reference-routed sparse self-attention and shot-wise cross-attention to route identities from the indexed frames while isolating per-shot textual conditions. Given the scarcity of high-quality open-source multi-shot data, we construct a dataset of over 350K film shots annotated with semantic state transitions, entity grounding, and multi-granularity requests. We further build CaT-MBench, a benchmark on held-out films with the same annotations. It gives every baseline its native format and counts cross-shot consistency only when a cut opens a new camera setup, preventing repeated framing from passing as consistency. On CaT-MBench, CaT-Movie attains the highest average score and the most consistent characters among all methods. The project page is available at https://jinplu.github.io/CaT-Movie/.
BibTeX
@misc{lu2026cat,
  author = {Jinpeng Lu and Haoyuan Shi and Yuanshen Guan and Jiyuan Xia and Bin Wu and Yinda Chen and Zhiwei Xiong},
  title = {{CaT-Movie: Balancing Global Planning and Local Rendering for Long Multi-Shot Video Generation}},
  year = {2026},
  eprint = {2611.25620},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25620},
}
Figure