Abstract
Video sequences contain rich dynamic patterns, such as dynamic texture patterns that exhibit stationarity in the temporal domain, and action patterns that are non-stationary in either spatial or temporal domain. We show that an energy-based spatial-temporal generative ConvNet can be used to model and synthesize dynamic patterns. The model defines a probability distribution on the video sequence, and the log probability is defined by a spatial-temporal ConvNet that consists of multiple layers of spatial-temporal filters to capture spatial-temporal patterns of different scales. The model can be learned from the training video sequences by an "analysis by synthesis" learning algorithm that iterates between synthesizing video sequences from the currently learned model and updating model parameters based on the difference between synthesized and observed sequences. We demonstrate realistic synthesis and the ability to learn from incomplete sequences.
Introduction
While deep learning has excelled in discriminative video tasks like action recognition, generative modeling for video synthesis remains a significant challenge. Popular frameworks like Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs) often rely on auxiliary models (discriminators or encoders) and lack an explicit probability density definition. This paper introduces an energy-based spatial-temporal generative Convolutional Neural Network (ConvNet) designed to explicitly model the complex spatial and temporal distributions of dynamic patterns, such as flowing water or fire, in a single, unified framework.
Methodology
The proposed model is an energy-based model (EBM) where the probability distribution of a video sequence is defined by an energy function. This function is parameterized by a bottom-up 3D ConvNet utilizing spatial-temporal filters.
Analysis by Synthesis
Training follows a Maximum Likelihood Estimation (MLE) approach via an alternating "analysis by synthesis" algorithm:
- Sampling (Synthesis): Fake video sequences are generated from the current model distribution using Langevin dynamics, a Markov Chain Monte Carlo (MCMC) method driven by the ConvNet's reconstruction error.
- Learning (Updating): ConvNet weights are updated via gradient ascent to minimize the divergence between synthesized and real training videos.
For stationary dynamic textures, the network employs filters convolutional in both space and time. For non-stationary actions, fully connected top layers capture global structures. Notably, the model can handle incomplete data by sampling only missing pixels during the Langevin step, enabling simultaneous learning and inpainting.
Experimental Results
The model was evaluated on the DynTex++ dataset and various internet videos. A key advantage is its data efficiency, often learning from a single video clip or a very small mini-batch.
Synthesis of Dynamic Textures
The model demonstrates high fidelity in synthesizing various dynamic textures. Quantitative evaluation on 17 dynamic textures showed the proposed model achieved an average Structural Similarity Index (SSIM) of 0.6020, significantly outperforming baselines like HOSVD (0.2577) and Doretto's LDS (0.5602).
Robustness and Inpainting
Beyond synthesis, the model excels at video recovery and inpainting. It achieved an average pixel recovery error of 5.30 on videos corrupted by 50% noise, outperforming 3D Markov Random Field baselines. It also successfully performed background inpainting, such as removing moving objects while hallucinating plausible background dynamics.
Conclusion
This energy-based spatial-temporal ConvNet provides a robust framework for modeling dynamic patterns without auxiliary networks. While it currently lacks explicit part-based decomposition for complex actions, its ability to learn explicit energy landscapes makes it highly effective for synthesis and inpainting. Future work may involve combining this EBM with top-down dynamic generators for applications in intuitive physics and image-based optimal control.
Summary updated