TL;DR
PYoCo (Preserve Your Own COrrelation) introduces a novel noise prior for video diffusion that captures the inherent temporal correlations between frames. By finetuning pre-trained image models with this correlated noise, the authors achieve state-of-the-art video synthesis quality with significantly higher efficiency—using a 10× smaller model and 14× less training time—than previous methods trained from scratch.
What Problem Does This Paper Address?
While text-to-image (T2I) diffusion models have reached incredible levels of photorealism, extending them to video (T2V) remains a massive challenge. The primary hurdles are the scarcity of high-quality, captioned video datasets compared to image datasets and the astronomical computational cost of training video models from scratch.
A common shortcut is to finetune a pre-trained image model on video data. However, standard diffusion models assume that the noise added to each frame is independent and identically distributed (i.i.d.). The authors discover a fundamental mismatch here: if you take a real video and "reverse" it into noise space using a T2I model, the resulting noise maps are not independent; they are highly correlated across time. Forcing a model to learn video dynamics using i.i.d. noise priors essentially forces it to "forget" the natural correlations it already understands about image structure, leading to jittery, inconsistent videos.
Methods
The core innovation is the Progressive Noise Model. Instead of sampling independent noise for every frame in a sequence, PYoCo generates noise autoregressively. The noise for frame $i$ is a linear combination of the noise from frame $i-1$ and a small amount of new, independent noise. This ensures that the initial "static" the model starts with already has a temporal structure that mimics real-world video.
Beyond the noise prior, the system uses a cascaded architecture:
- Base Model: Generates a low-resolution (64x64), low-frame-rate (16 frames) video.
- Temporal Interpolation: Fills in the gaps to increase the frame count (up to 76 frames).
- Spatial Super-Resolution: Two subsequent stages that upscale the video to 256x256 and finally 1024x1024.
The authors also employ joint image-video finetuning, training on both images and videos simultaneously to ensure the model doesn't lose the high-quality aesthetic knowledge it gained during its initial image-only training.
Datasets
- UCF-101: A standard action recognition dataset used here for small-scale ablation studies and unconditional generation benchmarks.
- MSR-VTT: A large-scale video benchmark used for evaluating zero-shot text-to-video performance.
- Large-scale Internal Dataset: For their flagship model, they used a massive collection of 1.2 billion text-image pairs and 22.5 million text-video pairs, filtered for high aesthetic quality.
Key Insights and Ideas
- The "Noise Prior Gap": The most significant finding is that the choice of noise prior is not just a technical detail—it's a bottleneck. Aligning the training noise distribution with the natural correlation of video frames (as seen in the t-SNE plots) is what allows for efficient knowledge transfer from image models to video models.
- Efficiency via Prior Knowledge: PYoCo proves that you don't need the biggest model to get the best results. By better utilizing pre-trained image weights, their 253M parameter model outperformed the 1.2B parameter Video Diffusion Model (VDM) on UCF-101.
- Preserving Artistic Style: Because PYoCo better preserves the image model's prior, it excels at generating videos in specific artistic styles (like Van Gogh paintings or watercolor) that are often missing from standard video datasets.
Key Results
- State-of-the-Art Performance: PYoCo achieved a zero-shot Inception Score (IS) of 47.76 on UCF-101, significantly higher than Make-A-Video (33.00) and Video LDM (33.45).
- Visual Quality: Qualitatively, the model produces videos with superior fine details and temporal consistency compared to major baselines like Imagen Video and Make-A-Video.
- Computational Savings: On the UCF-101 benchmark, PYoCo required only 75 GPU-days of training compared to 925 GPU-days for VDM, while achieving better scores.
Limitations and Future Work
The authors note that the hyperparameter $\alpha$ (which controls the strength of the correlation) requires careful tuning. If $\alpha$ is too high, the model produces "frozen" videos with almost no motion; if it's too low, the visual quality and diversity degrade.
While PYoCo handles short clips excellently, the paper does not deeply explore extremely long-term temporal consistency (e.g., minutes-long videos) or complex multi-step causal interactions, which remain open challenges for the broader field of video generation.
Summary updated