Your Image is Secretly the Last Frame of a Pseudo Video

Wenlong Chen, Wenlin Chen, Lapo Rastrelli, Yingzhen Li

Abstract

Diffusion models, which can be viewed as a special case of hierarchical variational autoencoders (HVAEs), have shown profound success in generating photo-realistic images. In contrast, standard HVAEs often produce images of inferior quality compared to diffusion models. In this paper, we hypothesize that the success of diffusion models can be partly attributed to the additional self-supervision information for their intermediate latent states provided by corrupted images, which along with the original image form a pseudo video. Based on this hypothesis, we explore the possibility of improving other types of generative models with such pseudo videos. Specifically, we first extend a given image generative model to their video generative model counterpart, and then train the video generative model on pseudo videos constructed by applying data augmentation to the original images. Furthermore, we analyze the potential issues of first-order Markov data augmentation methods, which are typically used in diffusion models, and propose to use more expressive data augmentation to construct more useful information in pseudo videos. Our empirical results on the CIFAR10 and CelebA datasets demonstrate that improved image generation quality can be achieved with additional self-supervised information from pseudo videos.

Introduction: The Identifiability Problem

In the landscape of generative AI, Diffusion Models currently outperform older architectures like Hierarchical Variational Autoencoders (HVAEs). This paper argues that the core difference lies in identifiability.

Comparison of HVAE and Diffusion Model latent states. HVAEs lack concrete targets for intermediate states, whereas Diffusion models use noisy images as self-supervision.

In a standard HVAE, intermediate latent states are unobserved, forcing the model to guess their structure within a vast hypothesis space. Diffusion models solve this by using "self-supervision": every intermediate step has a concrete target, typically a noisy version of the original image. The authors investigate whether this "secret sauce"—the use of intermediate corrupted states—can be applied to other generative frameworks to boost performance.

Methodology: Thinking in Pseudo Videos

The researchers introduce a framework that treats a single static image as the final frame of a pseudo video.

  1. Extension to Video Models: Existing image models are adapted into video counterparts. For instance, a standard VQ-VAE is extended to a C-ViViT (a video-based transformer), and an image-based GPT becomes VideoGPT.
  2. Pseudo Video Construction: Training data is created by taking a real image and applying recursive data augmentation, such as Gaussian blurring or noise, to create a sequence of "frames" leading up to the original image.
Example of a pseudo video sequence generated from a single image, showing the progression from a coarse, corrupted state to the final clear image.
  1. Higher-Order Markov Chains: While most diffusion models are "first-order Markov" (each step depends only on the previous one), the authors propose higher-order pseudo videos. Here, the model observes a window of multiple previous frames to predict the next. Mathematically, they prove (Theorem 4.1) that this reduces the lower bound of reconstruction error.

Experimental Setup

The models were evaluated on two standard computer vision benchmarks:

  • CIFAR-10: $32 \times 32$ pixel images across 10 categories.
  • CelebA: $64 \times 64$ pixel images of celebrity faces.

Key Insights

  • Self-Supervision as an Anchor: By providing "pseudo-frames" as targets, the model has a clear roadmap for image construction from coarse to fine detail, rather than wandering in an unconstrained latent space.
  • The "U-Turn" in Performance: There is a diminishing return effect. Moving from 1 frame to 8 frames provides a massive boost, but moving to 18 frames can sometimes make the task too complex, slightly degrading performance in certain metrics.
  • Markovian Limitations: The standard first-order approach in diffusion is not necessarily optimal. Allowing the model to see a longer history of the corruption process helps it better understand the underlying image structure.

Results

  • Reconstruction Quality: On CelebA, a 1-frame VQ-VAE achieved an FID of 24.62. Using an 18-frame pseudo video, the reconstruction FID dropped to 2.27, indicating significantly sharper images.
  • Generation Quality: For autoregressive generation (VideoGPT) on CIFAR-10, an 8-frame pseudo video improved the FID from 72.06 to 54.60.
  • Higher-Order Superiority: In video diffusion experiments on CIFAR-10, a 4-frame high-order model achieved an FID of 12.58, outperforming the first-order version's 17.30.

Limitations and Future Work

Training on pseudo videos introduces higher computational overhead and memory requirements. Additionally, the choice of augmentation (e.g., blur vs. noise) significantly impacts performance depending on the architecture. Future research may explore applying these pseudo-sequence principles to non-visual data types like text or molecular structures.

Summary updated

Abstract

Diffusion models, which can be viewed as a special case of hierarchical variational autoencoders (HVAEs), have shown profound success in generating photo-realistic images. In contrast, standard HVAEs often produce images of inferior quality compared to diffusion models. In this paper, we hypothesize that the success of diffusion models can be partly attributed to the additional self-supervision information for their intermediate latent states provided by corrupted images, which along with the original image form a pseudo video. Based on this hypothesis, we explore the possibility of improving other types of generative models with such pseudo videos. Specifically, we first extend a given image generative model to their video generative model counterpart, and then train the video generative model on pseudo videos constructed by applying data augmentation to the original images. Furthermore, we analyze the potential issues of first-order Markov data augmentation methods, which are typically used in diffusion models, and propose to use more expressive data augmentation to construct more useful information in pseudo videos. Our empirical results on the CIFAR10 and CelebA datasets demonstrate that improved image generation quality can be achieved with additional self-supervised information from pseudo videos.
BibTeX
@inproceedings{chen2025your,
  author = {Wenlong Chen and Wenlin Chen and Lapo Rastrelli and Yingzhen Li},
  title = {{Your Image is Secretly the Last Frame of a Pseudo Video}},
  booktitle = {DeLTa @ ICLR},
  year = {2025},
  url = {https://openreview.net/forum?id=ur8nvWUWr1},
}
Figure