TL;DR
This paper reveals that the secret to the success of diffusion models lies in treating image generation as a sequence of self-supervised steps—effectively turning a static image into the final frame of a "pseudo video." By extending standard generative models (like VQ-VAEs and autoregressive transformers) into video models trained on these artificial video trajectories, the authors significantly boost image generation quality. Furthermore, they prove that higher-order Markov chains provide a much more powerful self-supervision signal than the first-order chains typically used in standard diffusion.
What Problem Does This Paper Address?
Why do diffusion models drastically outperform traditional sequential generative models like Hierarchical Variational Autoencoders (HVAEs)? The authors point to a key structural difference: standard HVAEs have unobserved, highly flexible intermediate latent states, leading to an optimization nightmare known as unidentifiability—there are simply too many ways the model can choose to represent the data across its layers.
Diffusion models fix this by using a pre-defined, fixed corruption process (like adding Gaussian noise). This acts as an explicit self-supervision signal: at every step, the model knows exactly what corrupted version of the target image it is supposed to reconstruct.
The authors ask a compelling question: Can we inject this same flavor of trajectory-based self-supervision into other popular generative architectures to unlock similar performance gains?
Methods
The authors introduce a generalized framework to scale up standard image generative models by converting them into their video generative counterparts and training them on "pseudo videos."
- Pseudo Video Construction: A single target image is transformed into a sequence of frames using deterministic data augmentations (such as recursive blurring or noise addition). The sequence is ordered so that the early frames are highly corrupted, and the final frame is the clean, original image.
- Model Extension: They apply this recipe to two popular frameworks:
- VQ-VAE: They extend this to C-ViViT (a video transformer autoencoder) and use autoregressive (VideoGPT) or masked (Phenaki) transformers to model the latent space.
- DDPM (Diffusion): They extend an improved DDPM image model into a Video Diffusion Model that predicts frames autoregressively.
- Higher-Order Markov Chains: Crucially, the authors mathematically demonstrate that first-order Markov chains (where frame $t$ only depends on frame $t-1$, as in standard diffusion) are sub-optimal for autoregressive video generation. Because a first-order history provides no extra context, the video model gains no advantage from looking further back. To solve this, they introduce a high-order Markov strategy where noise is added to the rolling average of all previous frames, forcing the model to exploit long-term temporal context.
Datasets
The framework was evaluated on standard image synthesis benchmarks:
- CIFAR-10 ($32 \times 32$ images)
- CelebA ($64 \times 64$ face images)
- Binarized MNIST (used strictly for an initial proof-of-concept demonstrating the impact of a fixed heat-equation corruption process versus a learnable HVAE encoder).
Key Insights and Ideas
- Images as Destinations: Image generation doesn't have to happen in a single, unguided shot. Framing an image as the destination of a predictable trajectory (a pseudo video) provides a powerful inductive bias that simplifies the optimization landscape.
- The Autoregressive Flaw of First-Order Diffusion: While first-order Markov chains work beautifully for standard diffusion models, they fail when plugged directly into autoregressive video generation frameworks. Higher-order dependencies are required to truly leverage the power of sequence modeling architectures like Transformers or Video UNets.
- A New Dimension for Scaling: Instead of making image models deeper or wider by stacking layers, researchers can scale models dimensionally by transforming them into video architectures that learn from temporal context.
Key Results
- Sharper VQ-VAE Reconstructions: Incorporating pseudo videos drastically improved VQ-VAE/C-ViViT reconstruction quality. On CelebA, the reconstruction Frechet Inception Distance (FID) plummeted from 24.62 (1-frame baseline) to 2.27 (18-frame pseudo video).
- Improved Generation Metrics: For latent masked generation (Phenaki) on CelebA, the generation FID improved from 27.34 (1-frame) to 16.66 (18-frame).
- High-Order Superiority in Diffusion: In video diffusion experiments on CelebA, standard first-order Markov pseudo videos actually degraded performance (FID degraded to 13.61 from a 7.76 baseline). However, the proposed high-order Markov pseudo videos successfully improved the generation FID to 6.88.
Limitations and Future Work
- Diminishing Returns on Frame Count: Adding more frames isn't always better. On CIFAR-10, 8-frame pseudo videos outperformed 18-frame sequences. The authors note that longer pseudo videos create highly complex prior distributions that transformers struggle to capture perfectly.
- Massive Design Space: There are infinitely many ways to construct a pseudo video (different blending functions, noise schedules, or blurring kernels). Finding the optimal data augmentation strategy for a specific architecture remains an open empirical challenge.
- Beyond Vision: While creating pseudo videos via image corruption is intuitive, it remains unclear how to properly define and inject equivalent self-supervised "pseudo sequences" for non-visual modalities like text or molecular graphs.
Summary updated