Introduction
Generating high-quality video from text descriptions remains a significant challenge in computer vision. Unlike static image generation, video synthesis requires maintaining temporal consistency across frames while managing the high computational costs associated with 3D data. Traditional approaches often rely on 3D U-Net architectures or autoregressive frame-by-frame generation, both of which are memory-intensive and require vast amounts of paired text-video data.
Grid Diffusion Models (GDM) propose a novel solution by "flattening" the temporal dimension. By representing a sequence of frames as a single 2x2 "grid image," GDM leverages powerful pre-trained text-to-image (T2I) models like Stable Diffusion to generate consistent video with significantly reduced computational overhead and training data requirements.
Methodology: Flattening Time into Space
The core innovation of GDM is the Key Grid Image. Instead of treating video as a 3D volume, the model operates on a 512x512 image containing four 254x254 frames arranged in a 2x2 grid. This approach allows the model to utilize 2D spatial processing for temporal tasks.
Key Grid Image Generation
The process begins by fine-tuning a pre-trained Stable Diffusion model to generate these 2x2 grids from a text prompt. A key insight is that the Variational Autoencoder (VAE) in standard T2I models naturally captures global spatial structures. When trained on grid images, this latent space helps maintain visual consistency across the four sub-frames, effectively "freezing" the style and content across the temporal sequence.
Autoregressive Grid Image Interpolation
To transform these four keyframes into a fluid video, GDM employs a hierarchical interpolation stage. Using a "masked" grid image—where certain frames are omitted—the model is prompted to "fill in the blanks" based on the previous grid. This technique, inspired by image editing models like InstructPix2Pix, ensures smooth transitions between keyframes.
For longer sequences, the model generates subsequent key grid images autoregressively. By conditioning each new grid on the previous one, the model maintains narrative and visual continuity over extended durations.
Datasets and Evaluation
The model was trained on a highly efficient subset of the Webvid-10M dataset (only 0.1 million samples), demonstrating high data efficiency. Evaluation was conducted on standard benchmarks MSR-VTT and UCF-101. Additionally, the authors introduced CGCaption, a new benchmark consisting of 500 complex prompts generated by GPT-4, designed to test the model's ability to handle intricate, descriptive language.
Results and Performance
GDM demonstrates state-of-the-art performance on standard benchmarks while maintaining exceptional efficiency.
- Quality and Consistency: On the MSR-VTT benchmark, GDM achieved a Frechet Video Distance (FVD) of 375, a substantial improvement over existing models like VideoFusion (550) and CogVideo (1294).
- Computational Efficiency: The grid-based approach offers constant memory scaling. When generating 128-frame videos, GDM reduced GPU memory consumption by 74.08% compared to the FreeNoise baseline.
- Human Evaluation: In side-by-side user studies, GDM was consistently preferred over baselines across metrics including text alignment, video quality, and motion smoothness.
Discussion and Future Work
By treating time as a spatial component, GDM bypasses the data scarcity and computational hurdles of 3D video models. This "grid" philosophy also enables direct application of image-based tools, such as text-guided video editing and style transfer, to video sequences.
However, the model remains dependent on the capabilities of its underlying T2I base. Future work will explore denser grids (e.g., 4x4) for more complex motions and the integration of multi-modal inputs, such as synchronized audio generation.
Summary updated