TiVGAN: Text to Image to Video Generation with Step-by-Step Evolutionary Generator

Doyeon Kim, Donggyu Joo, Junmo Kim

Abstract

TiVGAN introduces a curriculum learning approach for text-to-video synthesis, evolving from a single high-quality image to a full-length video sequence. By prioritizing spatial details before temporal continuity, the model generates sharper, more text-accurate videos than previous end-to-end methods.

Introduction

Generating video from text descriptions remains a significant challenge in computer vision. Unlike static image generation, video synthesis requires both spatial "completeness" (high-quality individual frames) and temporal "connectivity" (smooth transitions). Existing models often struggle with stability and resolution, frequently resulting in blurry outputs or "mode collapse," where the model produces generic results regardless of the input. TiVGAN addresses these issues by stabilizing the training process through a staged evolutionary approach.

Methodology

The Text-to-Image-to-Video (TiVGAN) framework decomposes the generation process into two distinct stages:

  1. Text-to-Image Stage: The model first learns to generate a single high-quality frame matching the text description. A Gated Recurrent Unit (GRU) transforms text embeddings into a latent vector, which the generator then converts into an image.
  2. Evolutionary Generation Stage: Once the model masters single-frame generation, it is "evolved" to produce increasingly longer sequences (1 → 2 → 4 → 8 → 16 frames).
Overview of the TiVGAN evolutionary training process, showing the progression from single frames to full sequences.

To support this evolution, TiVGAN utilizes two key technical innovations:

  • Step-Discriminator Initialization: New discriminators are initialized with weights from previous steps, specifically dividing the first layer's weights by two to account for doubled input channels, ensuring a stable training signal.
  • Independent Samples Pairing (ISP): This technique forces the generator to produce diverse versions of a video for the same prompt using different random noises, preventing mode collapse.

Experiments and Results

TiVGAN was evaluated on three diverse benchmarks: KTH Action (human actions), MUG Facial Expression (emotions), and Kinetics (complex YouTube videos).

Example of generated facial expressions from the MUG dataset, demonstrating temporal consistency and visual quality.

The model consistently outperformed state-of-the-art baselines like TGAN and MoCoGAN across both qualitative and quantitative metrics. On the KTH Action dataset, TiVGAN achieved a Frechet Inception Distance (FID) of 47.34, a significant improvement over MoCoGAN’s 81.85. On the Kinetics dataset, the generated videos reached a 77.8% classification accuracy, remarkably close to the 78.1% accuracy of real videos.

Conclusion

TiVGAN demonstrates that a "spatial first, temporal second" approach significantly improves video synthesis quality. By anchoring the video in a strong initial frame, the model learns complex motions without losing fine spatial details. While currently limited by resolution and the computational cost of multi-phase training, TiVGAN provides a robust framework for stable, text-conditioned video generation.

Summary updated

BibTeX
@article{kim2020tivgan,
  author = {Doyeon Kim and Donggyu Joo and Junmo Kim},
  title = {{TiVGAN: Text to Image to Video Generation with Step-by-Step Evolutionary Generator}},
  journal = {IEEE Access},
  year = {2020},
  doi = {10.1109/access.2020.3017881},
}
Figure