TL;DR
Why do diffusion models outperform standard hierarchical VAEs when both generate images through sequential steps? This paper argues the secret lies in intermediate self-supervision: corrupted states provide explicit learning targets, effectively turning a static image into the finale of a "pseudo video." By expanding static image models into video architectures trained on synthetic frame sequences, the authors show that image generation quality can be substantially improved across multiple model families.
What Problem Does This Paper Address?
Sequential image generation models turn noise into images via multiple steps. Diffusion models and Hierarchical Variational Autoencoders (HVAEs) share a very similar sequential foundation, yet diffusion models consistently produce higher-fidelity images.
The authors identify a key structural difference:
- HVAEs treat intermediate latent states as unobserved variables to be learned end-to-end. This creates an unidentifiability issue—infinitely many intermediate representations can explain the data, creating an enormous search space filled with bad local optima.
- Diffusion models fix the intermediate states by corrupting the target image with known noise levels. This provides explicit, inductive self-supervision at every step.
This raises an intriguing question: Can we boost other image generative models by explicitly supervising their intermediate states using artificially created "pseudo videos"?
Methods
The proposed framework converts static training images into multi-frame pseudo videos by applying sequential corruptions (blurring or noise). The original image is always placed as the final frame ($x_T$), while earlier frames represent degraded precursors ($x_1, \dots, x_{T-1}$).
The authors apply this recipe across two model paradigms:
- Discrete Latent Models (VQ-VAE Counterparts):
- A 3D Video Autoencoder (C-ViViT) compresses blurred pseudo videos into discrete spatiotemporal tokens.
- Sequence priors are trained over these tokens using either autoregressive transformers (VideoGPT) or bidirectional masked prediction (Phenaki).
- Continuous Diffusion Models:
- An autoregressive video diffusion model predicts upcoming frames conditioned on preceding frames in a sliding context window.
- Higher-Order Markov Augmentation: The authors theoretically prove that standard first-order Markov chains (like standard diffusion forward paths, where $x_{t-1} \to x_{t-2}$) render past frames conditionally independent, providing zero extra predictive power to autoregressive context windows ($p(x_T \mid x_{T-1}, x_{T-2}) = p(x_T \mid x_{T-1})$). To fix this, they introduce higher-order transitions by corrupting the running mean of all preceding frames: $$x_{T-t} = \sqrt{1 - \beta_t} \left(\frac{1}{t}\sum_{s=1}^t x_{T-t+s}\right) + \sqrt{\beta_t} \epsilon$$
Datasets
- Binarized MNIST: Used as a proof of concept with heat-equation dissipation.
- CIFAR-10 ($32 \times 32$): Used for blur-based discrete token experiments and noise-based video diffusion.
- CelebA ($64 \times 64$): Evaluated across both discrete token and continuous diffusion setups. (No new external datasets were introduced; existing benchmarks were transformed into 4-, 8-, and 18-frame pseudo videos).
Key Insights and Ideas
- Temporal scaling beats spatial depth: Instead of stacking deeper layers into static image models (which often hits diminishing returns), extending the architecture across time with pseudo frames consistently improves reconstruction sharpness and generation fidelity.
- First-order Markov chains are mathematically suboptimal for autoregressive generation: Under non-parametric assumptions, the reconstruction error $\mathbb{E}[\|x_T - \hat{x}_T\|^2]$ strictly decreases with more context frames only if the conditional dependency $x_T \not\perp\!\!\!\perp x_{s_2} \mid x_{s_1}$ is preserved. First-order chains break this, causing autoregressive predictors to ignore older frames.
- Domain transfer of corruption physics: Intermediate states don't have to be Gaussian noise. Inverting heat dissipation equations or Gaussian blurs provides structured trajectories that guide models toward sharper edges.
Key Results
- Proof-of-Concept HVAE (MNIST): Fixing the encoder to an inverted heat equation boosted Inception Score from 7.27 (standard learnable HVAE) to 9.32.
- VQ-VAE / Phenaki Reconstruction & Generation:
- Reconstruction FID: On CelebA, 1-frame VQ-VAE achieved an FID of 24.62, which plunged to 5.72 with an 8-frame pseudo video and 2.27 with 18 frames.
- Masked Generation FID: On CIFAR-10, MaskGit (1-frame) reached 89.78 FID, whereas Phenaki trained on 8-frame blurred pseudo videos improved to 35.50. On CelebA, FID improved from 27.34 (1-frame) down to 16.66 (18-frame).
- Video Diffusion vs. Image DDPM:
- On CelebA, standard 1-frame DDPM scored 7.76 FID. First-order 4-frame video diffusion degraded performance to 13.61, but the proposed higher-order Markov pseudo video reached 6.88 FID.
- Increasing standard image DDPM to 4,000 diffusion steps yielded 7.87 FID on CelebA, proving the 4-frame video model's gain (at 3,000 total steps) was structural, not just an artifact of extra computation.
Limitations and Future Work
- Diminishing Returns with Frame Length: Performance does not always increase monotonically with sequence length. On CIFAR-10, 8-frame Phenaki outperformed 18-frame Phenaki (35.50 vs. 47.65 FID), likely because modeling the joint prior of long pseudo sequences becomes excessively difficult.
- Augmentation Sensitivity: Strategies do not transfer universally across architectures. While Gaussian blur worked best for discrete token models (C-ViViT), Gaussian noise performed significantly better than blur for video diffusion (12.58 vs. 15.33 FID on CIFAR-10).
- Search Space of Pseudo Videos: The design space of higher-order data augmentations and corruption dynamics is vast; finding theoretically optimal trajectories remains an open question.
Summary updated