Abstract
Dynamic patterns are characterized by complex spatial and motion patterns. Understanding dynamic patterns requires a disentangled representational model that separates the factorial components. A commonly used model for dynamic patterns is the state space model, where the state evolves over time according to a transition model and the state generates the observed image frames according to an emission model. To model the motions explicitly, it is natural for the model to be based on the motions or the displacement fields of the pixels. Thus in the emission model, we let the hidden state generate the displacement field, which warps the trackable component in the previous image frame to generate the next frame while adding a simultaneously emitted residual image to account for the change that cannot be explained by the deformation. The warping of the previous image is about the trackable part of the change of image frame, while the residual image is about the intrackable part of the image. We use a maximum likelihood algorithm to learn the model parameters that iterates between inferring latent noise vectors that drive the transition model and updating the parameters given the inferred latent vectors. Meanwhile we adopt a regularization term to penalize the norms of the residual images to encourage the model to explain the change of image frames by trackable motion. Unlike existing methods on dynamic patterns, we learn our model in unsupervised setting without ground truth displacement fields or optical flows. In addition, our model defines a notion of intrackability by the separation of warped component and residual component in each image frame. We show that our method can synthesize realistic dynamic pattern, and disentangling appearance, trackable and intrackable motions. The learned models can be useful for motion transfer, and it is natural to adopt it to define and measure intrackability of a dynamic pattern.
Introduction
Modeling dynamic patterns—such as flowing water, flickering flames, or human facial expressions—presents a significant challenge in computer vision. These patterns involve a complex interplay between static appearance and dynamic motion. Traditional generative models, such as Variational Autoencoders (VAEs) or Generative Adversarial Networks (GANs), often fail to explicitly disentangle these factors. Moreover, real-world motion is rarely purely predictable; it consists of "trackable" components (like the movement of a physical object) and "intrackable" components (like chaotic textures or lighting changes). This paper introduces a motion-based generative model that unsupervisedly separates videos into appearance, trackable motion (optical flow), and intrackable motion (residuals).
Proposed Methodology
The proposed model is a non-linear state-space model parameterized by neural networks. Unlike models that generate frames independently, this approach generates a sequence where each frame is derived from the previous one through a combination of warping and residual addition.
As illustrated in Figure 1, the model follows a sequential process:
- Appearance Generation: An initial base image ($I_0$) is generated from a latent appearance vector.
- Trackable Motion: At each time step $t$, a hidden state $s_t^M$ generates a 2D displacement field $M_t$. This field is used to warp the previous frame $I_{t-1}$ to produce the trackable component of the next frame.
- Intrackable Motion: Simultaneously, a hidden state $s_t^R$ generates a residual image $R_t$. This residual accounts for changes that cannot be explained by pixel displacement, such as chaotic splashing or lighting shifts.
The model is trained using Maximum Likelihood Estimation (MLE) with Langevin dynamics for latent vector inference. An $L_2$ regularization term is applied to the residual images to force the model to prioritize trackable motion explanations.
Experimental Results
The model was evaluated on several datasets, including DynTex++ for dynamic textures and the MUG dataset for facial expressions, without any ground-truth motion annotations.
Synthesis and Realism
A human perceptual study was conducted to evaluate the realism of the synthesized patterns. Participants compared the model's output against several baselines, including MoCoGAN and TwoStream.
As shown in Figure 5, the proposed model maintains a higher level of perceived realism as observation time increases, significantly outperforming other methods. This suggests that the disentangled representation captures the underlying dynamics more effectively than entangled models.
Motion Transfer and Intrackability
The separation of motion and appearance allows for zero-shot motion transfer. For instance, facial expressions extracted from one individual can be applied to static images of entirely different domains, such as animals or cartoon characters, without further training. Additionally, the model provides a mathematical metric for "intrackability" by calculating the ratio of the residual image's norm to the total image norm, offering a formal way to quantify video complexity.
Conclusion
This work demonstrates that unsupervised disentanglement of appearance and motion is achievable through a motion-based generative framework. By explicitly modeling trackable and intrackable components, the model achieves state-of-the-art synthesis quality and enables novel applications like cross-domain motion transfer. Future work will focus on incorporating geometric deformation models to handle unaligned subjects, further enhancing the model's flexibility.
Summary updated