A Survey: Spatiotemporal Consistency in Video Generation

Zhiyu Yin, Kehai Chen, Xuefeng Bai, Ruili Jiang, Juntao Li, Hongdong Li, Jin Liu, Yang Xiang, Jun Yu, Min Zhang

TL;DR

This survey paper reframes video generation as a sequential sampling process from a high-dimensional spatiotemporal distribution, focusing heavily on the critical challenge of maintaining spatiotemporal consistency. It provides a structured taxonomy covering generation models, feature representations, frameworks, training strategies, and evaluation metrics, mapping out the current landscape and future frontiers of AI-generated video content.

What Problem Does This Paper Address?

While static image generation (via models like Midjourney or DALL-E) has matured significantly, video generation introduces a massive hurdle: spatiotemporal consistency. A generated video isn't just a collection of high-quality independent images; the primary subjects must retain their identity, the background shouldn't morph randomly (spatial consistency), and the motion between frames must be physically plausible and smooth without flickering or sudden jumps (temporal consistency).

The authors address the lack of a unified, systematic review focusing specifically on this core consistency bottleneck. They categorize common failures—such as object teleportation, lighting flicker, and action sequence violations—and organize the vast literature around how modern architectures attempt to solve them.

Methods

The paper organizes the technical landscape into several key pillars designed to preserve consistency:

  1. Generation Models: It compares the foundational mathematical principles, advantages, and disadvantages of Variational Autoencoders (VAEs), Autoregressive (AR) models, Diffusion Models (DMs), and Flow Models (FMs).
  2. Feature Representations: It details how video data is compressed and tokenized. Key approaches include causal representations (ensuring one-way transmission of information to prevent future frame leakage), time-space decoupling, and content-motion decoupling.
  3. Generation Frameworks: It analyzes how models structure the sampling process, comparing joint spatiotemporal optimization in Diffusion Transformers (DiTs) against sequential conditional sampling in autoregressive frameworks, multi-stage cascading, and interactive generation.
  4. Training & Post-processing Strategies: It reviews training paradigms like progressive learning (scaling from short, low-res clips to long, high-res videos), joint image-video learning, and post-hoc refinement techniques (e.g., frame interpolation, stabilization, and deblurring).
Fig. 1. Overview Structure Diagram.

Datasets

As a survey paper, it does not introduce a new dataset but provides a comprehensive categorization of existing benchmarks used to evaluate spatiotemporal capabilities:

  • Text Alignment Benchmarks: Evaluating how well video progression matches textual narratives (e.g., StoryBench, StoryEval).
  • Temporal Dynamics Benchmarks: Focusing on temporal coherence, logic, and motion naturalness (e.g., MiraBench, DEVIL, T2VBench, VMBench).
  • Comprehensive Performance Benchmarks: Evaluating visual quality, text alignment, and motion holistically (e.g., EvalCrafter, AIGCBench, Video-Bench, VBench).

Key Insights and Ideas

  • Video as Conditional Transition Probability: The core conceptual shift is viewing video generation as learning a conditional distribution $p(x_t | x_{<t}, C)$, where $C$ represents control information. Temporal consistency is fundamentally a transition probability modeling problem.
  • The Model Trade-off Matrix: Autoregressive models offer strong causal consistency but suffer from slow inference and cumulative errors. Diffusion models achieve state-of-the-art results through iterative global optimization, while Flow Models show promise for local smoothness but struggle with complex long-term dependencies.
  • The Power of Decoupling: To maintain consistency without blowing up computational budgets, top-tier models increasingly separate features into static content (low-frequency global structures) and dynamic motion (high-frequency details), or utilize multi-scale feature pyramids.
Fig. 2. Feature Representations.

Key Results

The paper provides a massive meta-analysis of over 50 prominent video generation models spanning from 2021 to 2025 (including OpenAI's Sora, Meta's Movie Gen, CogVideoX, and HunyuanVideo).

The clear architectural trend shows the field converging toward Diffusion Transformers (DiT) combined with 3D Causal VAEs. This combination allows models to compress video efficiently while enforcing causal temporal relationships, resulting in the highest levels of spatiotemporal consistency observed in commercial and open-source models today.

Limitations and Future Work

Despite rapid progress, the authors highlight several critical bottlenecks that remain unresolved:

  • Long Video Generation: Consistently preserving character identity and environmental states across thousands of frames or multiple scene transitions still causes severe computational memory strain and training instability.
  • Fine-Grained Personalization: Aligning multi-modal signals (like text combined with specific facial expressions or sketches) often results in localized distortions or control signals fading over long durations.
  • Higher-Level Consistency: Current models struggle with "Video Emotion Expression" (sustaining a psychological atmosphere or logical character mood swings across shots) and simulating complex physical world interactions ("Video World Models").
  • Flawed Evaluation Metrics: Most metrics are migrated from static image generation (like FID) and fail to assess the video as a temporal whole. The field urgently needs automated benchmarks that directly quantify long-range identity preservation and narrative coherence.

Summary updated

Abstract

Video generation aims to produce temporally coherent sequences of visual frames, representing a pivotal advancement in Artificial Intelligence Generated Content (AIGC). Compared to static image generation, video generation poses unique challenges: it demands not only high-quality individual frames but also strong temporal coherence to ensure consistency throughout the spatiotemporal sequence. Although research addressing spatiotemporal consistency in video generation has increased in recent years, systematic reviews focusing on this core issue remain relatively scarce. To fill this gap, this paper views the video generation task as a sequential sampling process from a high-dimensional spatiotemporal distribution, and further discusses spatiotemporal consistency. We provide a systematic review of the latest advancements in the field. The content spans multiple dimensions including generation models, feature representations, generation frameworks, post-processing techniques, training strategies, benchmarks and evaluation metrics, with a particular focus on the mechanisms and effectiveness of various methods in maintaining spatiotemporal consistency. Finally, this paper explores future research directions and potential challenges in this field, aiming to provide valuable insights for advancing video generation technology. The project link is https://github.com/Yin-Z-Y/A-Survey-Spatiotemporal-Consistency-in-Video-Generation.
BibTeX
@misc{yin2025survey,
  author = {Zhiyu Yin and Kehai Chen and Xuefeng Bai and Ruili Jiang and Juntao Li and Hongdong Li and Jin Liu and Yang Xiang and Jun Yu and Min Zhang},
  title = {{A Survey: Spatiotemporal Consistency in Video Generation}},
  year = {2025},
  eprint = {2502.17863},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.17863},
}
Figure