Enabling the Encoder-Empowered GAN-based Video Generators for Long Video Generation

Jingbo Yang, Adrian G. Bors

Abstract

Despite the remarkable progress in the video generation field, generating videos of longer-term remains challenging due to the challenge of sustaining the temporal consistency and continuity in the resulting synthesized movement while ensuring realism. In this paper, we propose a recall mechanism for enabling an encoder-empowered short-term video generator to produce long-term videos. This mechanism connects smoothly short video clips by modeling their temporal connections. We propose the Recall Encoder-GAN3 (REncGAN3), which enables an Encoder-based Generative Adversarial Network (GAN) to connect short generated video clips into longer sequences of hundreds of frames. The recall mechanism, defined through a loss function, enables an appropriate plasticity-continuity balance in the resulting long video stream. The proposed long-term video generation method ensures the generation of several hundred frames displaying consistent movement, which is non-repetitive while the computational memory costs are similar to those of short video generation models.
BibTeX
@inproceedings{yang2023enabling,
  author = {Jingbo Yang and Adrian G. Bors},
  title = {{Enabling the Encoder-Empowered GAN-based Video Generators for Long Video Generation}},
  booktitle = {ICIP},
  year = {2023},
  doi = {10.1109/icip49359.2023.10222725},
}

Loading related content...

Figure