Grid Diffusion Models for Text-to-Video Generation

Taegyeong Lee, Soyeong Kwon, Taehwan KimCVPR 2024 · DOI · arXiv

Introduction

Generating high-quality video from text descriptions remains a significant challenge in computer vision. Unlike static image generation, video synthesis requires maintaining temporal consistency across frames while managing the high computational costs associated with 3D data. Traditional approaches often rely on 3D U-Net architectures or autoregressive frame-by-frame generation, both of which are memory-intensive and require vast amounts of paired text-video data.

Grid Diffusion Models (GDM) propose a novel solution by "flattening" the temporal dimension. By representing a sequence of frames as a single 2x2 "grid image," GDM leverages powerful pre-trained text-to-image (T2I) models like Stable Diffusion to generate consistent video with significantly reduced computational overhead and training data requirements.

Methodology: Flattening Time into Space

The core innovation of GDM is the Key Grid Image. Instead of treating video as a 3D volume, the model operates on a 512x512 image containing four 254x254 frames arranged in a 2x2 grid. This approach allows the model to utilize 2D spatial processing for temporal tasks.

Overview of the Grid Diffusion Model pipeline, including key grid generation and autoregressive interpolation.

Key Grid Image Generation

The process begins by fine-tuning a pre-trained Stable Diffusion model to generate these 2x2 grids from a text prompt. A key insight is that the Variational Autoencoder (VAE) in standard T2I models naturally captures global spatial structures. When trained on grid images, this latent space helps maintain visual consistency across the four sub-frames, effectively "freezing" the style and content across the temporal sequence.

Autoregressive Grid Image Interpolation

To transform these four keyframes into a fluid video, GDM employs a hierarchical interpolation stage. Using a "masked" grid image—where certain frames are omitted—the model is prompted to "fill in the blanks" based on the previous grid. This technique, inspired by image editing models like InstructPix2Pix, ensures smooth transitions between keyframes.

Detailed view of the multi-step interpolation process used to expand keyframes into a full video sequence.

For longer sequences, the model generates subsequent key grid images autoregressively. By conditioning each new grid on the previous one, the model maintains narrative and visual continuity over extended durations.

Datasets and Evaluation

The model was trained on a highly efficient subset of the Webvid-10M dataset (only 0.1 million samples), demonstrating high data efficiency. Evaluation was conducted on standard benchmarks MSR-VTT and UCF-101. Additionally, the authors introduced CGCaption, a new benchmark consisting of 500 complex prompts generated by GPT-4, designed to test the model's ability to handle intricate, descriptive language.

Results and Performance

GDM demonstrates state-of-the-art performance on standard benchmarks while maintaining exceptional efficiency.

  • Quality and Consistency: On the MSR-VTT benchmark, GDM achieved a Frechet Video Distance (FVD) of 375, a substantial improvement over existing models like VideoFusion (550) and CogVideo (1294).
  • Computational Efficiency: The grid-based approach offers constant memory scaling. When generating 128-frame videos, GDM reduced GPU memory consumption by 74.08% compared to the FreeNoise baseline.
  • Human Evaluation: In side-by-side user studies, GDM was consistently preferred over baselines across metrics including text alignment, video quality, and motion smoothness.

Discussion and Future Work

By treating time as a spatial component, GDM bypasses the data scarcity and computational hurdles of 3D video models. This "grid" philosophy also enables direct application of image-based tools, such as text-guided video editing and style transfer, to video sequences.

However, the model remains dependent on the capabilities of its underlying T2I base. Future work will explore denser grids (e.g., 4x4) for more complex motions and the integration of multi-modal inputs, such as synchronized audio generation.

Summary updated

BibTeX
@inproceedings{lee2024grid,
  author = {Taegyeong Lee and Soyeong Kwon and Taehwan Kim},
  title = {{Grid Diffusion Models for Text-to-Video Generation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.00834},
}
Figure