When prompting a modern text-to-video or image-to-video model to generate a complex scene—such as a flower blooming or a dog running—the results are often visually stunning but frustratingly still. Instead of fluid, cinematic motion, the model frequently outputs what resembles a high-quality slideshow or a static image with minor camera jitter.
This phenomenon, variously termed static video bias, motion stagnation, or temporal stagnation, represents a fundamental bottleneck in generative video modeling. While models have mastered spatial fidelity (generating beautiful individual frames), they consistently struggle with temporal dynamics (generating meaningful change between frames). Understanding why this happens requires looking at the mathematical shortcuts and data biases that influence their training.
Why Models Choose Stagnation
To a video diffusion model, stagnation is not a failure; it is an optimized shortcut. Several distinct mechanisms drive models toward this static local minimum.
The Lazy Loss Function
Standard diffusion models minimize training error by comparing predicted pixels to ground-truth pixels. However, as Shijie Wang et al. observe in their work on MotiF: Making Text Count in Image Animation with Motion Focal Loss, the vast majority of pixels in a typical training video remain entirely unchanged from frame to frame. A model can drastically reduce its training loss simply by duplicating the initial frame across the entire temporal sequence. This creates a powerful local minimum where the network is penalized far less for failing to generate motion than for generating motion with minor spatial distortions.
Conditional Image Leakage
In image-to-video (I2V) tasks, this problem is compounded by the presence of a clean reference frame. During the early stages of the reverse diffusion process, when the latent space consists mostly of random noise, the model naturally over-relies on this clear conditional signal.
As detailed by the authors of Identifying and Solving Conditional Image Leakage in Image-to-Video Diffusion Model, this "conditional image leakage" causes the model to prioritize the structural stability of the reference frame over the temporal evolution required for realistic motion. As a result, the requested dynamics vanish, a phenomenon also documented by Jie Tian et al. in Extrapolating and Decoupling Image-to-Video Generation Models: Motion Modeling is Easier Than You Think.
The Human Preference Trap
One might expect reinforcement learning or Direct Preference Optimization (DPO) to correct this bias, but standard preference tuning often exacerbates it. As Ziyi Wu et al. demonstrate in DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Models, high-motion video generations are inherently more prone to visual artifacts, warping, and tearing. Human annotators consistently favor clean, low-motion videos over dynamic ones that contain glitches. Consequently, preference optimization inadvertently teaches models that the safest path to a high score is to eliminate motion entirely.
A similar trade-off occurs in customized video generation (e.g., animating a specific user-provided subject). As Hengjia Li et al. in MagicID: Hybrid Preference Optimization for ID-Consistent and Dynamic-Preserved Video Customization and Xiangtai Li et al. in MotionBooth: Motion-Aware Customized Text-to-Video Generation note, minimizing movement is the easiest way for a model to preserve subject identity and avoid "identity drifting."
Data Distribution Deficits
Finally, the underlying data itself is heavily biased. A significant portion of web-scale video datasets consists of static or low-motion shots, as highlighted by Xuan Ju et al. in MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captions and Yousef Yeganeh et al. in VISAGE: Video Synthesis using Action Graphs for Surgery. Furthermore, when datasets do contain camera movements, they often feature static environments, leading models to falsely associate camera motion with an unnaturally frozen world, as shown by Ivan Skorokhodov et al. in AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformers.
Mechanisms for Unlocking Motion
To break free from the static trap, researchers have developed several distinct strategies that force models to prioritize temporal change over frame replication.
Standard Diffusion: [Frame 1] ---> [Copy Frame 1] ---> [Copy Frame 1] (Low Loss, No Motion)
Motion-Aware: [Frame 1] ---> [Apply Flow] ---> [Evolved Frame] (High Energy, Dynamic)
1. Regulating Temporal Attention
During generation, a model's temporal attention maps dictate how information flows across frames. In static videos, these maps flatten out, focusing heavily on the initial frame and exhibiting low high-frequency energy.
To counteract this, training-free methods like those proposed by Jiazi Bu et al. in ByTheWay: Boost Your Text-to-Video Generation Model to Higher Quality in a Training-free Way and Yumeng Li et al. in VSTAR: Generative Temporal Nursing for Longer Dynamic Video Synthesis explicitly regularize temporal attention maps during inference. By amplifying high-frequency energy and forcing cross-frame interaction, these techniques prevent the model from collapsing into a sequence of identical frames.
2. Modifying Noise Schedules and Initializations
Another approach targets the early-stage dominance of the conditional image. The authors of Identifying and Solving Conditional Image Leakage in Image-to-Video Diffusion Model introduced a mathematically optimized starting point (Analytic-Init) alongside a time-dependent noise schedule (TimeNoise) to disrupt the model's over-reliance on the static reference frame.
Similarly, Ludan Ruan et al. in UniVG: Towards UNIfied-modal Video Generation utilized Biased Gaussian Noise (BGN) to explicitly control and encourage motion intensity, preventing motion stagnation by shifting the noise distribution to favor frame-to-frame variance.
3. Motion-Aware Loss Functions
Instead of treating all pixels equally, models can be forced to care about the regions that should be moving. The Motion Focal Loss introduced by Shijie Wang et al. uses optical flow to identify dynamic regions in the ground-truth video and heavily weights the loss on those moving pixels. This prevents the static background pixels from dominating the gradient, forcing the model to learn the actual transformations requested by text prompts.
4. Fine-Grained Preference Optimization
To fix the human preference trap, Ziyi Wu et al. developed DenseDPO, a segment-level preference optimization framework. By using guided sampling to generate video pairs that share identical motion trajectories but differ only in fine details, they decoupled motion from artifact generation. This allows human annotators to reward visual quality without penalizing motion intensity, effectively stripping away the static bias learned during standard preference tuning.
Evaluating True Metamorphosis
As the field moves away from static outputs, evaluating a model's capacity for true temporal evolution has become critical. Traditional benchmarks often fail to distinguish between high frame-level quality and actual motion.
To address this, benchmarks like ChronoMagic-Bench, developed by Shenghai Yuan et al. in ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generation, focus specifically on "metamorphic" evaluation. By testing a model's ability to depict long-term physical transformations—such as biological growth, melting, or time-lapse changes—rather than merely adding minor camera jitter to a still scene, these new evaluation frameworks ensure that future video models generate true physical evolution rather than high-fidelity stagnation.
Key Foundational Work
Video Prediction with Appearance and Motion Conditions (2018) — contributed 1 concept elements to this direction.
Large Language Models are Frame-level Directors for Zero-shot Text-to-Video Generation (2023) — contributed 1 concept elements to this direction.