Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models

Songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang, Ming-Yu Liu, Yogesh Balaji

TL;DR

PYoCo (Preserve Your Own COrrelation) introduces a novel noise prior for video diffusion that captures the inherent temporal correlations between frames. By finetuning pre-trained image models with this correlated noise, the authors achieve state-of-the-art video synthesis quality with significantly higher efficiency—using a 10× smaller model and 14× less training time—than previous methods trained from scratch.

What Problem Does This Paper Address?

While text-to-image (T2I) diffusion models have reached incredible levels of photorealism, extending them to video (T2V) remains a massive challenge. The primary hurdles are the scarcity of high-quality, captioned video datasets compared to image datasets and the astronomical computational cost of training video models from scratch.

A common shortcut is to finetune a pre-trained image model on video data. However, standard diffusion models assume that the noise added to each frame is independent and identically distributed (i.i.d.). The authors discover a fundamental mismatch here: if you take a real video and "reverse" it into noise space using a T2I model, the resulting noise maps are not independent; they are highly correlated across time. Forcing a model to learn video dynamics using i.i.d. noise priors essentially forces it to "forget" the natural correlations it already understands about image structure, leading to jittery, inconsistent videos.

Figure 2: Visualizing the noise map correlations. (a) visualizes the t-SNE plot of the noise maps corresponding to input frames randomly sampled from videos. These noise maps are obtained by running a diffusion ODE [49, 48] on the input frames using a trained text-to-image model, but in the opposite direction of image synthesis (σ : 0 →σmax). The green dots in the background denote the reference noise maps sampled from an i.i.d. Gaussian distribution. The red dots and yellow dots are noise maps corresponding to input frames coming from different videos. We found they are spread out and share no correlation. On the other hand, the noise maps corresponding to the frames coming from the same video (shown in blue dots) are clustered together. (b) Using an i.i.d. noise model (orange dots) for finetuning text-to-image models for video synthesis is not ideal since temporal correlations between frames are not modeled. To remedy this, we propose a progressive noise model in which the correlation between different noise maps is injected along the temporal axis. Our progressive noise model (blue dots) aptly models the correlations present in the video noise maps.

Methods

The core innovation is the Progressive Noise Model. Instead of sampling independent noise for every frame in a sequence, PYoCo generates noise autoregressively. The noise for frame $i$ is a linear combination of the noise from frame $i-1$ and a small amount of new, independent noise. This ensures that the initial "static" the model starts with already has a temporal structure that mimics real-world video.

Beyond the noise prior, the system uses a cascaded architecture:

  1. Base Model: Generates a low-resolution (64x64), low-frame-rate (16 frames) video.
  2. Temporal Interpolation: Fills in the gaps to increase the frame count (up to 76 frames).
  3. Spatial Super-Resolution: Two subsequent stages that upscale the video to 256x256 and finally 1024x1024.
The PYoCo cascaded architecture, moving from a base model through temporal interpolation to high-resolution spatial upscaling.

The authors also employ joint image-video finetuning, training on both images and videos simultaneously to ensure the model doesn't lose the high-quality aesthetic knowledge it gained during its initial image-only training.

Datasets

  • UCF-101: A standard action recognition dataset used here for small-scale ablation studies and unconditional generation benchmarks.
  • MSR-VTT: A large-scale video benchmark used for evaluating zero-shot text-to-video performance.
  • Large-scale Internal Dataset: For their flagship model, they used a massive collection of 1.2 billion text-image pairs and 22.5 million text-video pairs, filtered for high aesthetic quality.

Key Insights and Ideas

  • The "Noise Prior Gap": The most significant finding is that the choice of noise prior is not just a technical detail—it's a bottleneck. Aligning the training noise distribution with the natural correlation of video frames (as seen in the t-SNE plots) is what allows for efficient knowledge transfer from image models to video models.
  • Efficiency via Prior Knowledge: PYoCo proves that you don't need the biggest model to get the best results. By better utilizing pre-trained image weights, their 253M parameter model outperformed the 1.2B parameter Video Diffusion Model (VDM) on UCF-101.
  • Preserving Artistic Style: Because PYoCo better preserves the image model's prior, it excels at generating videos in specific artistic styles (like Van Gogh paintings or watercolor) that are often missing from standard video datasets.

Key Results

  • State-of-the-Art Performance: PYoCo achieved a zero-shot Inception Score (IS) of 47.76 on UCF-101, significantly higher than Make-A-Video (33.00) and Video LDM (33.45).
  • Visual Quality: Qualitatively, the model produces videos with superior fine details and temporal consistency compared to major baselines like Imagen Video and Make-A-Video.
  • Computational Savings: On the UCF-101 benchmark, PYoCo required only 75 GPU-days of training compared to 925 GPU-days for VDM, while achieving better scores.
Qualitative comparison showing PYoCo producing better details and photorealism than Make-A-Video and Imagen Video.

Limitations and Future Work

The authors note that the hyperparameter $\alpha$ (which controls the strength of the correlation) requires careful tuning. If $\alpha$ is too high, the model produces "frozen" videos with almost no motion; if it's too low, the visual quality and diversity degrade.

While PYoCo handles short clips excellently, the paper does not deeply explore extremely long-term temporal consistency (e.g., minutes-long videos) or complex multi-step causal interactions, which remain open challenges for the broader field of video generation.

Summary updated

Abstract

Despite tremendous progress in generating high-quality images using diffusion models, synthesizing a sequence of animated frames that are both photorealistic and temporally coherent is still in its infancy. While off-the-shelf billion-scale datasets for image generation are available, collecting similar video data of the same scale is still challenging. Also, training a video diffusion model is computationally much more expensive than its image counterpart. In this work, we explore finetuning a pretrained image diffusion model with video data as a practical solution for the video synthesis task. We find that naively extending the image noise prior to video noise prior in video diffusion leads to sub-optimal performance. Our carefully designed video noise prior leads to substantially better performance. Extensive experimental validation shows that our model, Preserve Your Own Correlation (PYoCo), attains SOTA zero-shot text-to-video results on the UCF-101 and MSR-VTT benchmarks. It also achieves SOTA video generation quality on the small-scale UCF-101 benchmark with a $10\times$ smaller model using significantly less computation than the prior art.
BibTeX
@inproceedings{ge2023preserve,
  author = {Songwei Ge and Seungjun Nah and Guilin Liu and Tyler Poon and Andrew Tao and Bryan Catanzaro and David Jacobs and Jia-Bin Huang and Ming-Yu Liu and Yogesh Balaji},
  title = {{Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
}
Figure