Abstract
TiVGAN introduces a curriculum learning approach for text-to-video synthesis, evolving from a single high-quality image to a full-length video sequence. By prioritizing spatial details before temporal continuity, the model generates sharper, more text-accurate videos than previous end-to-end methods.
Introduction
Generating video from text descriptions remains a significant challenge in computer vision. Unlike static image generation, video synthesis requires both spatial "completeness" (high-quality individual frames) and temporal "connectivity" (smooth transitions). Existing models often struggle with stability and resolution, frequently resulting in blurry outputs or "mode collapse," where the model produces generic results regardless of the input. TiVGAN addresses these issues by stabilizing the training process through a staged evolutionary approach.
Methodology
The Text-to-Image-to-Video (TiVGAN) framework decomposes the generation process into two distinct stages:
- Text-to-Image Stage: The model first learns to generate a single high-quality frame matching the text description. A Gated Recurrent Unit (GRU) transforms text embeddings into a latent vector, which the generator then converts into an image.
- Evolutionary Generation Stage: Once the model masters single-frame generation, it is "evolved" to produce increasingly longer sequences (1 → 2 → 4 → 8 → 16 frames).
To support this evolution, TiVGAN utilizes two key technical innovations:
- Step-Discriminator Initialization: New discriminators are initialized with weights from previous steps, specifically dividing the first layer's weights by two to account for doubled input channels, ensuring a stable training signal.
- Independent Samples Pairing (ISP): This technique forces the generator to produce diverse versions of a video for the same prompt using different random noises, preventing mode collapse.
Experiments and Results
TiVGAN was evaluated on three diverse benchmarks: KTH Action (human actions), MUG Facial Expression (emotions), and Kinetics (complex YouTube videos).
The model consistently outperformed state-of-the-art baselines like TGAN and MoCoGAN across both qualitative and quantitative metrics. On the KTH Action dataset, TiVGAN achieved a Frechet Inception Distance (FID) of 47.34, a significant improvement over MoCoGAN’s 81.85. On the Kinetics dataset, the generated videos reached a 77.8% classification accuracy, remarkably close to the 78.1% accuracy of real videos.
Conclusion
TiVGAN demonstrates that a "spatial first, temporal second" approach significantly improves video synthesis quality. By anchoring the video in a strong initial frame, the model learns complex motions without losing fine spatial details. While currently limited by resolution and the computational cost of multi-phase training, TiVGAN provides a robust framework for stable, text-conditioned video generation.
Summary updated