Your Image is Secretly the Last Frame of a Pseudo Video

Wenlong Chen, Wenlin Chen, Lapo Rastrelli, Yingzhen Li

TL;DR

This paper reveals that the secret to the success of diffusion models lies in treating image generation as a sequence of self-supervised steps—effectively turning a static image into the final frame of a "pseudo video." By extending standard generative models (like VQ-VAEs and autoregressive transformers) into video models trained on these artificial video trajectories, the authors significantly boost image generation quality. Furthermore, they prove that higher-order Markov chains provide a much more powerful self-supervision signal than the first-order chains typically used in standard diffusion.

What Problem Does This Paper Address?

Why do diffusion models drastically outperform traditional sequential generative models like Hierarchical Variational Autoencoders (HVAEs)? The authors point to a key structural difference: standard HVAEs have unobserved, highly flexible intermediate latent states, leading to an optimization nightmare known as unidentifiability—there are simply too many ways the model can choose to represent the data across its layers.

Diffusion models fix this by using a pre-defined, fixed corruption process (like adding Gaussian noise). This acts as an explicit self-supervision signal: at every step, the model knows exactly what corrupted version of the target image it is supposed to reconstruct.

Figure 1: Diffusion model vs HVAE: compared to standard HVAEs, diffusion models incorporate inductive bias for its intermediate latent states with self-supervised signals.

The authors ask a compelling question: Can we inject this same flavor of trajectory-based self-supervision into other popular generative architectures to unlock similar performance gains?

Methods

The authors introduce a generalized framework to scale up standard image generative models by converting them into their video generative counterparts and training them on "pseudo videos."

  1. Pseudo Video Construction: A single target image is transformed into a sequence of frames using deterministic data augmentations (such as recursive blurring or noise addition). The sequence is ordered so that the early frames are highly corrupted, and the final frame is the clean, original image.
  2. Model Extension: They apply this recipe to two popular frameworks:
  3. VQ-VAE: They extend this to C-ViViT (a video transformer autoencoder) and use autoregressive (VideoGPT) or masked (Phenaki) transformers to model the latent space.
  4. DDPM (Diffusion): They extend an improved DDPM image model into a Video Diffusion Model that predicts frames autoregressively.
  5. Higher-Order Markov Chains: Crucially, the authors mathematically demonstrate that first-order Markov chains (where frame $t$ only depends on frame $t-1$, as in standard diffusion) are sub-optimal for autoregressive video generation. Because a first-order history provides no extra context, the video model gains no advantage from looking further back. To solve this, they introduce a high-order Markov strategy where noise is added to the rolling average of all previous frames, forcing the model to exploit long-term temporal context.
Figure 3: An example of pseudo video constructed by transforming an image of a dog using blurring.

Datasets

The framework was evaluated on standard image synthesis benchmarks:

  • CIFAR-10 ($32 \times 32$ images)
  • CelebA ($64 \times 64$ face images)
  • Binarized MNIST (used strictly for an initial proof-of-concept demonstrating the impact of a fixed heat-equation corruption process versus a learnable HVAE encoder).

Key Insights and Ideas

  • Images as Destinations: Image generation doesn't have to happen in a single, unguided shot. Framing an image as the destination of a predictable trajectory (a pseudo video) provides a powerful inductive bias that simplifies the optimization landscape.
  • The Autoregressive Flaw of First-Order Diffusion: While first-order Markov chains work beautifully for standard diffusion models, they fail when plugged directly into autoregressive video generation frameworks. Higher-order dependencies are required to truly leverage the power of sequence modeling architectures like Transformers or Video UNets.
  • A New Dimension for Scaling: Instead of making image models deeper or wider by stacking layers, researchers can scale models dimensionally by transforming them into video architectures that learn from temporal context.

Key Results

  • Sharper VQ-VAE Reconstructions: Incorporating pseudo videos drastically improved VQ-VAE/C-ViViT reconstruction quality. On CelebA, the reconstruction Frechet Inception Distance (FID) plummeted from 24.62 (1-frame baseline) to 2.27 (18-frame pseudo video).
  • Improved Generation Metrics: For latent masked generation (Phenaki) on CelebA, the generation FID improved from 27.34 (1-frame) to 16.66 (18-frame).
  • High-Order Superiority in Diffusion: In video diffusion experiments on CelebA, standard first-order Markov pseudo videos actually degraded performance (FID degraded to 13.61 from a 7.76 baseline). However, the proposed high-order Markov pseudo videos successfully improved the generation FID to 6.88.
Figure 4: Ground-truth images and reconstructed images from VQVAE/CViViT trained on CI- FAR10.

Limitations and Future Work

  • Diminishing Returns on Frame Count: Adding more frames isn't always better. On CIFAR-10, 8-frame pseudo videos outperformed 18-frame sequences. The authors note that longer pseudo videos create highly complex prior distributions that transformers struggle to capture perfectly.
  • Massive Design Space: There are infinitely many ways to construct a pseudo video (different blending functions, noise schedules, or blurring kernels). Finding the optimal data augmentation strategy for a specific architecture remains an open empirical challenge.
  • Beyond Vision: While creating pseudo videos via image corruption is intuitive, it remains unclear how to properly define and inject equivalent self-supervised "pseudo sequences" for non-visual modalities like text or molecular graphs.

Summary updated

Abstract

Diffusion models, which can be viewed as a special case of hierarchical variational autoencoders (HVAEs), have shown profound success in generating photo-realistic images. In contrast, standard HVAEs often produce images of inferior quality compared to diffusion models. In this paper, we hypothesize that the success of diffusion models can be partly attributed to the additional self-supervision information for their intermediate latent states provided by corrupted images, which along with the original image form a pseudo video. Based on this hypothesis, we explore the possibility of improving other types of generative models with such pseudo videos. Specifically, we first extend a given image generative model to their video generative model counterpart, and then train the video generative model on pseudo videos constructed by applying data augmentation to the original images. Furthermore, we analyze the potential issues of first-order Markov data augmentation methods, which are typically used in diffusion models, and propose to use more expressive data augmentation to construct more useful information in pseudo videos. Our empirical results on the CIFAR10 and CelebA datasets demonstrate that improved image generation quality can be achieved with additional self-supervised information from pseudo videos.
BibTeX
@inproceedings{chen2025your,
  author = {Wenlong Chen and Wenlin Chen and Lapo Rastrelli and Yingzhen Li},
  title = {{Your Image is Secretly the Last Frame of a Pseudo Video}},
  booktitle = {DeLTa @ ICLR},
  year = {2025},
  url = {https://openreview.net/forum?id=ur8nvWUWr1},
}
Figure