Introduction
Generating video using Generative Adversarial Networks (GANs) presents a significant challenge compared to static image generation. Video data requires models to simultaneously capture high-dimensional spatial structures and complex temporal dynamics. Most conventional approaches attempt to train these components jointly, which frequently leads to optimization difficulties such as "mode collapse"—where the generator produces limited, repetitive outputs—or complete training divergence. This paper addresses these issues by proposing a decoupled framework that disentangles frame content ("what") from motion ("how").
Methods
The authors introduce a two-step training scheme that conceptualizes video generation as a navigation problem on a learned image manifold.
Step 1: Frame Generation ($G_F$)
First, a convolutional generator ($G_F$) is trained to produce high-quality static frames. To ensure diversity and stability, the authors employ a multi-discriminator approach using 48 discriminators. Each discriminator evaluates a different low-dimensional random projection of the image, preventing the GAN from collapsing into a single image type.
Step 2: Sequence Generation ($G_V$)
Once $G_F$ is proficient, its weights are frozen. A recurrent model, specifically a bi-directional LSTM, is then trained to generate a sequence of latent vectors ($z_{F1}, z_{F2}, \dots, z_{FN}$). These vectors act as coordinates within the pre-trained $G_F$ latent space, effectively defining trajectories that correspond to natural-looking motion.
Experimental Setup
The framework was evaluated using the Bouncing Balls dataset, a standard benchmark for temporal modeling. The training set consisted of 100,000 samples of 30-frame videos. Experiments included both 1-ball and 3-ball variants to test the model's ability to handle varying complexity and its capacity to generalize temporal laws across different frame contents.
Results and Discussion
The decoupled approach yielded significant improvements in stability and quality.
- Temporal Coherence: The model successfully learned to "navigate" the complex, non-continuous manifold of valid frames. Quantitative analysis using Mean Squared Error (MSE) between consecutive frames showed that for the 1-ball case, generated videos achieved an MSE of 0.0227, nearly matching the real data's 0.0222.
- Diversity and Stability: The multi-discriminator setup effectively mitigated strong mode collapse, producing a wide variety of trajectories. The two-step process proved much more stable than end-to-end training, requiring less hyperparameter tuning.
- Content-Dynamics Disentanglement: In a key experiment, a temporal model trained on 3 balls was paired with a frame generator trained on 1 ball. The resulting video featured a single ball moving with the complex dynamics of the 3-ball system, demonstrating that the model successfully learned independent temporal laws.
Conclusion
While the model successfully avoided total mode collapse, the authors noted instances of "partial mode collapse" and occasional physical inconsistencies, such as balls changing trajectory without impact. Future work will explore increasing the number of discriminators during temporal training to further enhance diversity and applying this manifold navigation approach to other time-series domains like audio or sensor data.
Summary updated