Abstract
Dynamic patterns are characterized by complex spatial and motion patterns. Understanding dynamic patterns requires a disentangled representational model that separates the factorial components. A commonly used model for dynamic patterns is the state space model, where the state evolves over time according to a transition model and the state generates the observed image frames according to an emission model. To model the motions explicitly, it is natural for the model to be based on the motions or the displacement fields of the pixels. Thus in the emission model, we let the hidden state generate the displacement field, which warps the trackable component in the previous image frame to generate the next frame while adding a simultaneously emitted residual image to account for the change that cannot be explained by the deformation. The warping of the previous image is about the trackable part of the change of image frame, while the residual image is about the intrackable part of the image. We use a maximum likelihood algorithm to learn the model that iterates between inferring latent noise vectors that drive the transition model and updating the parameters given the inferred latent vectors. Meanwhile we adopt a regularization term to penalize the norms of the residual images to encourage the model to explain the change of image frames by trackable motion. Unlike existing methods on dynamic patterns, we learn our model in unsupervised setting without ground truth displacement fields. In addition, our model defines a notion of intrackability by the separation of warped component and residual component in each image frame. We show that our method can synthesize realistic dynamic pattern, and disentangling appearance, trackable and intrackable motions. The learned models are useful for motion transfer, and it is natural to adopt it to define and measure intrackability of a dynamic pattern.
Introduction
Modeling dynamic patterns—such as the flickering of a flame, the flow of a waterfall, or the nuances of human facial expressions—remains a significant challenge in computer vision. These patterns involve a complex interplay between static appearance and temporal changes. Traditional generative models often fail to separate what an object looks like from how it moves, leading to "entangled" representations. Furthermore, many existing approaches rely on supervised data, such as pre-calculated optical flow, to understand motion.
This paper introduces a motion-based generative model designed to disentangle appearance, trackable motion (pixel displacement), and intrackable motion (stochastic changes like lighting or complex textures) in an entirely unsupervised manner.
Methodology
The proposed framework utilizes a State Space Model (SSM) where the hidden state $s_t$ at each time step is partitioned into two components: a motion state $s^M_t$ and a residual state $s^R_t$.
Emission Model and Motion Decomposition
The core of the model lies in its emission process. The motion state $s^M_t$ generates a displacement field that "warps" the previous image frame to produce the trackable portion of the next frame. Simultaneously, the residual state $s^R_t$ generates a residual image to account for "intrackable" changes—elements like the glow of fire or the emergence of new ripples that cannot be explained by simple pixel deformation.
Learning and Regularization
The model is trained using a maximum likelihood algorithm. Instead of relying on standard GAN or VAE architectures, the authors employ Langevin dynamics to infer the latent noise vectors that drive the transition model. To ensure the model prioritizes trackable motion, a regularization term is applied to penalize the norm of the residual images, forcing the generator to explain as much temporal change as possible through warping.
Experiments and Results
The model was evaluated on diverse datasets, including DynTex++ (dynamic textures) and the MUG Facial Expression Dataset.
Disentanglement and Motion Transfer
A key strength of the model is its ability to disentangle motion from appearance without ground truth flow data. This enables "zero-shot" motion transfer, where a learned motion (e.g., a specific facial expression) can be applied to a completely different appearance (e.g., a different person or a cartoon character).
Perceptual Realism and Intrackability
In human subject tests, the model's synthesized videos were rated as more realistic than those from established baselines like MoCoGAN and LDS. Additionally, the authors introduced a quantitative metric for intrackability, defined as the ratio of the residual image's norm to the total image's norm. This allows for the objective measurement of a dynamic pattern's complexity; for instance, boiling water exhibits higher intrackability than a waving flag.
Conclusion
The motion-based generator model successfully achieves unsupervised disentanglement of appearance and motion components. While highly effective for aligned objects, the model currently faces limitations with large-scale geometric transformations or significant viewpoint changes. Future work involves integrating dedicated geometry generators to handle unaligned videos and 3D rotations.
Summary updated