Introduction
Generating realistic video is significantly more complex than generating static images. A successful model must not only synthesize a plausible subject (appearance) but also ensure that the subject moves naturally (motion) and maintains identity throughout the sequence (spatio-temporal consistency). Most existing video GANs suffer from "entanglement," where attempting to modify motion (e.g., making a person smile) inadvertently alters appearance (e.g., changing facial features). G3AN addresses this challenge by learning to separate these factors from raw video data without relying on pre-defined priors like keypoints or optical flow.
The Three-Stream Generator
The core innovation of G3AN is its Three-Stream Generator, which decomposes the generation process into three parallel paths:
- Spatial Stream ($G_S$): Focuses on appearance, processing a noise vector $z_a$ to define static features.
- Temporal Stream ($G_T$): Focuses on motion, processing a noise vector $z_m$ to define how features evolve.
- Video Stream ($G_V$): The primary stream that fuses the spatial and temporal components using Factorized (1+2)D Transposed Convolutions.
By splitting complex 3D operations into simpler 1D (temporal) and 2D (spatial) steps, the model becomes easier to optimize and produces higher visual quality. This architectural choice forces the separation of motion and appearance, preventing motion features from "leaking" into appearance features.
Factorized Spatio-temporal Self-Attention (F-SA)
To capture long-range dependencies across a video without the prohibitive computational cost of 3D self-attention, G3AN introduces Factorized Spatio-temporal Self-Attention (F-SA). This mechanism splits attention into a temporal-wise step followed by a spatial-wise step. This factorization allows the model to maintain global consistency across the entire video sequence while remaining computationally efficient enough for standard hardware.
Experimental Evaluation
G3AN was evaluated on four diverse datasets: MUG (facial expressions), UvA-NEMO (smiling), Weizmann (human actions), and UCF101 (complex "in-the-wild" actions).
Key Results
G3AN established new state-of-the-art benchmarks across multiple metrics:
- Quantitative Performance: The model achieved the lowest Fréchet Inception Distance (FID) on all datasets. On the MUG dataset, G3AN reached an FID of 67.12, a substantial improvement over MoCoGAN's 87.11.
- Human Evaluation: In blind side-by-side tests, human raters preferred G3AN-generated videos over MoCoGAN's approximately 84% of the time.
- Latent Space Manipulation: The architecture successfully disentangles factors, allowing researchers to hold a subject's identity constant while changing their expression, or change identity (e.g., adding a mustache) while keeping movement identical.
Conclusion
G3AN demonstrates that architectural disentanglement and factorization are powerful tools for video synthesis. By separating motion and appearance into distinct streams and factorizing expensive 3D operations, the model achieves superior visual quality and temporal consistency. While current experiments were conducted at 64x64 resolution, the authors aim to scale the model to higher resolutions and more complex, unconstrained environments in future work.
Summary updated