TL;DR
This survey paper reframes video generation as a sequential sampling process from a high-dimensional spatiotemporal distribution, focusing heavily on the critical challenge of maintaining spatiotemporal consistency. It provides a structured taxonomy covering generation models, feature representations, frameworks, training strategies, and evaluation metrics, mapping out the current landscape and future frontiers of AI-generated video content.
What Problem Does This Paper Address?
While static image generation (via models like Midjourney or DALL-E) has matured significantly, video generation introduces a massive hurdle: spatiotemporal consistency. A generated video isn't just a collection of high-quality independent images; the primary subjects must retain their identity, the background shouldn't morph randomly (spatial consistency), and the motion between frames must be physically plausible and smooth without flickering or sudden jumps (temporal consistency).
The authors address the lack of a unified, systematic review focusing specifically on this core consistency bottleneck. They categorize common failures—such as object teleportation, lighting flicker, and action sequence violations—and organize the vast literature around how modern architectures attempt to solve them.
Methods
The paper organizes the technical landscape into several key pillars designed to preserve consistency:
- Generation Models: It compares the foundational mathematical principles, advantages, and disadvantages of Variational Autoencoders (VAEs), Autoregressive (AR) models, Diffusion Models (DMs), and Flow Models (FMs).
- Feature Representations: It details how video data is compressed and tokenized. Key approaches include causal representations (ensuring one-way transmission of information to prevent future frame leakage), time-space decoupling, and content-motion decoupling.
- Generation Frameworks: It analyzes how models structure the sampling process, comparing joint spatiotemporal optimization in Diffusion Transformers (DiTs) against sequential conditional sampling in autoregressive frameworks, multi-stage cascading, and interactive generation.
- Training & Post-processing Strategies: It reviews training paradigms like progressive learning (scaling from short, low-res clips to long, high-res videos), joint image-video learning, and post-hoc refinement techniques (e.g., frame interpolation, stabilization, and deblurring).
Datasets
As a survey paper, it does not introduce a new dataset but provides a comprehensive categorization of existing benchmarks used to evaluate spatiotemporal capabilities:
- Text Alignment Benchmarks: Evaluating how well video progression matches textual narratives (e.g., StoryBench, StoryEval).
- Temporal Dynamics Benchmarks: Focusing on temporal coherence, logic, and motion naturalness (e.g., MiraBench, DEVIL, T2VBench, VMBench).
- Comprehensive Performance Benchmarks: Evaluating visual quality, text alignment, and motion holistically (e.g., EvalCrafter, AIGCBench, Video-Bench, VBench).
Key Insights and Ideas
- Video as Conditional Transition Probability: The core conceptual shift is viewing video generation as learning a conditional distribution $p(x_t | x_{<t}, C)$, where $C$ represents control information. Temporal consistency is fundamentally a transition probability modeling problem.
- The Model Trade-off Matrix: Autoregressive models offer strong causal consistency but suffer from slow inference and cumulative errors. Diffusion models achieve state-of-the-art results through iterative global optimization, while Flow Models show promise for local smoothness but struggle with complex long-term dependencies.
- The Power of Decoupling: To maintain consistency without blowing up computational budgets, top-tier models increasingly separate features into static content (low-frequency global structures) and dynamic motion (high-frequency details), or utilize multi-scale feature pyramids.
Key Results
The paper provides a massive meta-analysis of over 50 prominent video generation models spanning from 2021 to 2025 (including OpenAI's Sora, Meta's Movie Gen, CogVideoX, and HunyuanVideo).
The clear architectural trend shows the field converging toward Diffusion Transformers (DiT) combined with 3D Causal VAEs. This combination allows models to compress video efficiently while enforcing causal temporal relationships, resulting in the highest levels of spatiotemporal consistency observed in commercial and open-source models today.
Limitations and Future Work
Despite rapid progress, the authors highlight several critical bottlenecks that remain unresolved:
- Long Video Generation: Consistently preserving character identity and environmental states across thousands of frames or multiple scene transitions still causes severe computational memory strain and training instability.
- Fine-Grained Personalization: Aligning multi-modal signals (like text combined with specific facial expressions or sketches) often results in localized distortions or control signals fading over long durations.
- Higher-Level Consistency: Current models struggle with "Video Emotion Expression" (sustaining a psychological atmosphere or logical character mood swings across shots) and simulating complex physical world interactions ("Video World Models").
- Flawed Evaluation Metrics: Most metrics are migrated from static image generation (like FID) and fail to assess the video as a temporal whole. The field urgently needs automated benchmarks that directly quantify long-range identity preservation and narrative coherence.
Summary updated