Abstract
This paper studies the dynamic generator model for spatial-temporal processes such as dynamic textures and action sequences in video data. In this model, each time frame of the video sequence is generated by a generator model, which is a non-linear transformation of a latent state vector, where the non-linear transformation is parametrized by a top-down neural network. The sequence of latent state vectors follows a non-linear auto-regressive model, where the state vector of the next frame is a non-linear transformation of the state vector of the current frame as well as an independent noise vector that provides randomness in the transition. The non-linear transformation of this transition model can be parametrized by a feedforward neural network. We show that this model can be learned by an alternating back-propagation through time algorithm that iteratively samples the noise vectors and updates the parameters in the transition model and the generator model. We show that our training method can learn realistic models for dynamic textures and action patterns.
Introduction
Modeling spatial-temporal processes like dynamic textures (e.g., flowing water, fire) and action sequences in videos is a major challenge in computer vision. Recently, deep learning approaches have tackled this using Generative Adversarial Networks (GANs) or Variational Autoencoders (VAEs). However, these methods require auxiliary networks: GANs need a discriminator to judge realism, and VAEs need a complex inference network to guess the latent variables.
This paper proposes a fundamental shift: learning a powerful, non-linear dynamic model on its own, without these extra networks. The authors aim to build a model that can be trained end-to-end using a more direct maximum likelihood approach, simplifying the architecture and reducing the parameter count.
Methods
The proposed "dynamic generator model" consists of two main components:
- Transition Model: A feedforward neural network that acts as a non-linear auto-regressive model. It predicts the next latent state vector based on the current state and an independent Gaussian noise vector, which injects randomness into the motion.
- Emission Model: A top-down deconvolutional neural network that maps the latent state vector into a high-dimensional video frame.
To train this model without auxiliary networks, the authors introduce Alternating Back-Propagation Through Time (ABPTT). This algorithm alternates between two steps:
- Inferential BPTT: Given the current model parameters and the observed video, it uses Langevin dynamics to sample the sequence of latent noise vectors. The gradient of the log-posterior is calculated by back-propagating through time.
- Learning BPTT: Given these inferred noise vectors, it updates the network weights via gradient ascent to maximize the likelihood of the observed data.
To manage memory constraints during training, the model utilizes truncated back-propagation through time, processing the video in chunks. Additionally, a static "appearance vector" is introduced alongside the dynamic latent state to effectively disentangle content from motion.
Key Results
The model was evaluated on several datasets, including DynTex++, UCF101, and the MUG Facial Expression Dataset, for tasks ranging from texture synthesis to video inpainting.
Dynamic Texture Synthesis
In human perceptual tests (measuring the "fooling rate"), the model generated more realistic dynamic textures than baselines like MoCoGAN and TwoStream across various exposure times. Qualitative results demonstrate the model's ability to capture complex motions in fire, water, and flags.
As shown in the quantitative analysis below, the proposed method maintains a significantly higher fooling rate than competing models as exposure time increases, indicating superior temporal consistency and realism.
Action Generation and Inpainting
On a 9-class subset of UCF101, the model achieved an Inception score of 8.21 ± 0.09, significantly outperforming MoCoGAN (4.40) and TGAN (5.48). Furthermore, because the model explicitly infers latent variables to explain observed pixels, it naturally handles missing data. When trained on videos with 50% missing frames, the model successfully recovered the sequences, achieving an average pixel error of 5.04, beating the STGCN baseline (6.37).
Conclusion
This work demonstrates that simplicity is viable: the adversarial game of GANs or the lower-bound approximations of VAEs are not strictly necessary to train complex video generators. Direct maximum likelihood estimation, powered by Langevin dynamics for inference, provides a robust framework for spatial-temporal modeling. Future work may involve extending the model to generate physical variables like velocity fields or optical flows, creating a more physically meaningful "mental physics" model.
Summary updated