TL;DR
CaT-Movie is a framework for long-form, multi-shot video generation that maintains character and scene consistency across extended durations. By decoupling global narrative planning (via an LLM-driven storyboard) from local rendering (using identity-routing attention), it generates coherent movies that successfully transition between diverse camera setups without losing visual identity.
What Problem Does This Paper Address?
Generating a single high-quality video clip is now common, but stitching them into a coherent "movie" remains a massive challenge. Current multi-shot generators suffer from Coherence across Time (CaT) issues.
There are two main failure modes:
- Identity Drift: Characters or scenes gradually change their appearance as the video progresses across different generation windows.
- Copied Framing: To maintain consistency, some models "cheat" by repeating the exact same camera angle and background from previous shots, failing to produce the dynamic transitions (cuts) expected in cinema.
Existing benchmarks often reward this "cheating" because they measure consistency through simple feature similarity, which is highest when the frames are nearly identical.
Methods
CaT-Movie splits the task into two specialized components:
- Global Story Planner: A training-free module that uses a Large Language Model (Qwen3.8-27B) to expand a user's prompt into a structured JSON storyboard. This storyboard tracks entities, locations, camera settings, and "state" (e.g., what a character is wearing or holding). Deterministic code then maintains this state across shots, ensuring that if a character picks up a hat in Shot 1, they are still holding it in Shot 5.
- Local Window Renderer: A video diffusion transformer (fine-tuned from Wan2.2) that renders shots in small windows. It uses Reference-Routed Sparse Self-Attention to "borrow" identity features from specific "anchor views" (historical frames of the same character) indexed by the planner. This allows the model to maintain character likeness while the Shot-wise Cross-Attention ensures each shot follows its unique textual description and camera setup.
Datasets
The authors addressed the lack of high-quality multi-shot data by creating:
- Film Storyboard Dataset: A massive collection of over 350,000 film shots annotated with entity registries, semantic state transitions, and grounding (boxes/masks).
- CaT-MBench: A new benchmark specifically designed to test long-term consistency. Its key innovation is the "cut gate": a consistency check only counts as a "pass" if the model also successfully executed a planned camera setup change. This prevents models that simply "copy-paste" previous frames from scoring well.
Key Insights and Ideas
- The "Cut Gate" Philosophy: True cinematic consistency isn't just about things looking the same; it's about things looking the same from a different perspective. By gating evaluation on successful cuts, the authors force the model to learn identity, not just pixels.
- LLM as a "State Machine": Instead of asking the video model to remember everything, the LLM acts as a persistent memory of the "world state," which is then fed to the renderer as precise instructions.
- Anchor Views over Recency: Most memory-augmented models look at the most recent frames. CaT-Movie looks for the best frames (clearest view, latest attire change) of a specific entity, regardless of how far back they occurred.
Key Results
- Breaking the Pareto Frontier: As shown in Figure 4, CaT-Movie is the only method that achieves both high transition control (executing planned cuts) and high inter-shot consistency.
- Quantitative Dominance: On CaT-MBench, CaT-Movie achieved the highest average score (0.597) and the most consistent characters (0.651) among all tested methods, including StoryMem and HoloCine.
- Cut Execution: It successfully executed 97.4% of planned cuts, significantly outperforming other joint generators that often "merge" shots to maintain consistency.
Limitations and Future Work
- Instruction Following: While the model is excellent at persisting a state, it sometimes struggles to establish a new requested change (like a character suddenly putting on a hat) if the base video model's prior is too strong.
- Computational Overhead: Running a 27B parameter LLM for planning and a 5B parameter transformer for rendering is computationally intensive.
- Future Directions: The authors suggest that better integration of 3D-aware priors could further improve consistency when camera angles change drastically (e.g., a 180-degree reverse shot).
Summary updated