Abstract
This paper studies the dynamic generator model for spatialtemporal processes such as dynamic textures and action sequences in video data. In this model, each time frame of the video sequence is generated by a generator model, which is a non-linear transformation of a latent state vector, where the non-linear transformation is parametrized by a top-down neural network. The sequence of latent state vectors follows a non-linear auto-regressive model, where the state vector of the next frame is a non-linear transformation of the state vector of the current frame as well as an independent noise vector that provides randomness in the transition. The non-linear transformation of this transition model can be parametrized by a feedforward neural network. We show that this model can be learned by an alternating back-propagation through time algorithm that iteratively samples the noise vectors and updates the parameters in the transition model and the generator model. We show that our training method can learn realistic models for dynamic textures and action patterns.
Introduction
Modeling spatial-temporal processes, such as video sequences, presents a dual challenge: capturing the intricate visual appearance of individual frames while simultaneously modeling the complex, non-linear dynamics of their evolution over time. Traditional approaches like Linear Dynamic Systems (LDS) are often too simplistic for chaotic phenomena like waterfalls or nuanced human movement. While modern deep learning has introduced Generative Adversarial Networks (GANs) and Variational Auto-Encoders (VAEs), these methods often require auxiliary networks—discriminators for GANs or inference networks for VAEs—which can introduce training instabilities or increased complexity.
This research introduces a "one-piece" dynamic generator model that learns to synthesize complex video sequences using only its own internal gradients, bypassing the need for separate auxiliary networks.
Methodology
The proposed Dynamic Generator Model integrates two primary neural components:
- Transition Model: A feedforward network that predicts the subsequent latent state based on the current state and a Gaussian noise (innovation) vector, capturing the underlying motion logic.
- Emission Model: A top-down deconvolutional network that transforms the latent state into a high-dimensional image frame.
Training is achieved via Alternating Back-Propagation through Time (ABPTT), which avoids external discriminators or encoders. The process alternates between two steps:
- Inferential Back-Prop: Utilizing Langevin dynamics, the model "searches" for the sequence of noise vectors that best explains the observed video by flowing gradients backward from pixels through the emission and transition models.
- Learning Back-Prop: With the noise vectors estimated, the model updates its parameters to maximize the likelihood of the observed data.
This approach treats video as an inherently causal process, where each frame depends on its predecessor, facilitating effective future frame prediction and unsupervised inpainting.
Experiments and Results
The model was evaluated across diverse datasets, including DynTex++ (dynamic textures), the Weizmann Action Dataset (human actions), UCF101 (large-scale action recognition), and various animal action datasets.
Perceptual and Quantitative Performance
The model demonstrates exceptional perceptual realism. In human "fooling" tests, where participants were asked to distinguish between real and synthesized videos, the proposed method significantly outperformed baselines like LDS and MoCoGAN.
Quantitatively, the model achieved an Inception Score of 8.21 on the UCF101 dataset, nearly doubling the performance of MoCoGAN (4.40) and substantially exceeding TGAN (5.48). The robustness of the model is further evidenced by its performance in user error tests across varying exposure times, consistently maintaining a higher fool rate than competing methods.
Unsupervised Inpainting
Beyond synthesis, the model's understanding of video "logic" allows for robust unsupervised inpainting. It can successfully recover video data where 50% of frames are missing or large regions are occluded. Notably, when a moving object is masked, the model naturally fills in the background, as it cannot find a consistent motion logic for the masked area that aligns with the surrounding scene.
Conclusion
The Dynamic Generator Model trained via ABPTT offers a parameter-efficient and stable alternative to GAN-based video synthesis. By leveraging Langevin dynamics for inference, the model effectively uses its own gradients to navigate the latent space. While the iterative nature of Langevin dynamics can be computationally intensive compared to single-pass inference networks, the resulting perceptual quality and causal consistency are significant. Future work may involve extending this framework to controlled dynamic generators, allowing for interactive simulations or robotic planning by incorporating external action vectors.
Summary updated