VDSM: Unsupervised Video Disentanglement With State-Space Modeling and Deep Mixtures of Experts

Matthew J. Vowels, Necati Cihan Camgöz, Richard Bowden

Abstract

VDSM (Video Disentanglement via State-Space Modeling) is a fully unsupervised framework designed to decompose video sequences into three independent components: identity, pose, and dynamics. By utilizing structural inductive biases, such as a Mixture of Experts (MoE) decoder and hierarchical latent variables, VDSM achieves state-of-the-art performance in video generation and action transfer tasks without the need for manual labels or adversarial training.

Introduction

In the field of computer vision, disentanglement refers to the learning of representations where distinct physical factors—such as a subject's identity versus their specific expression or movement—are isolated into independent variables. This capability is essential for applications like action transfer, where the motion of one subject is mapped onto the appearance of another.

Achieving disentanglement in a purely unsupervised manner is theoretically challenging and typically requires specific "inductive biases" within the model architecture. While existing methods often rely on expensive human-labeled data or unstable Generative Adversarial Networks (GANs), VDSM proposes a solution that leverages the inherent structure of video data to achieve clean disentanglement, avoiding the complexities of supervision and GAN training.

Methodology

VDSM is built upon a Variational Autoencoder (VAE) backbone, enhanced with several structural innovations:

Factorized Latent Space

The model partitions the latent space into three distinct factors:

  1. Static Factor ($s$): Captures identity and content that remains constant throughout the video.
  2. Pose Factor ($z_t$): Represents time-varying appearance at each specific frame.
  3. Dynamics Factor ($d$): A global vector summarizing the overall motion or action sequence.

Mixture of Experts (MoE) Decoder

To prevent identity information from leaking into motion variables, VDSM employs a bank of smaller decoders. The model learns to blend these "experts" based on the identity factor ($s$), forcing the network to specialize its decoding process for specific identities.

Hierarchical State-Space Model

A transition network predicts the subsequent pose ($z_{t+1}$) conditioned on the current pose and the global dynamics factor ($d$), effectively modeling the temporal flow of the video.

Two-Stage Training

Training is conducted in two phases to prevent the model from finding "shortcuts" that bypass disentanglement:

  • Stage 1: Focuses on separating identity from pose using shuffled, non-sequential image groups. Temporal averaging is used to ensure the identity factor remains constant.
  • Stage 2: Learns sequential dynamics while the identity and pose encoders remain frozen.

Experimental Setup

The performance of VDSM was evaluated across four diverse benchmarks: the MUG Facial Expression Database, Colored Sprites, Moving MNIST, and Synthetic Swinging Pendula. These datasets test the model's ability to handle human faces, animated characters, and physical motion.

Example of a character identity from the Colored Sprites dataset.

Results and Discussion

VDSM demonstrates significant improvements over previous state-of-the-art methods like S3VAE and DSA.

Quantitative Performance

On the MUG dataset, VDSM achieved a Fréchet Inception Distance (FID) of 17.64, significantly outperforming the GAN-based G3AN (67.12), indicating superior visual realism. In action transfer tasks, VDSM improved accuracy consistency by over 30% compared to its closest competitors.

Qualitative Insights

The model successfully performs complex tasks such as swapping facial expressions between individuals and transferring animations (e.g., walking or slashing) between different cartoon sprites with minimal visual artifacts.

A different character identity performing a similar pose, illustrating disentanglement.

Key Insights

  • Structure over Supervision: By architectural design—such as forcing the identity factor to be a temporal average—the model is compelled to segregate constant features from dynamic ones.
  • Stability: Utilizing a VAE-based approach avoids the common pitfalls of GAN training, such as mode collapse, leading to a more robust and easier-to-train model.

Conclusion

While VDSM shows remarkable success, it currently requires a two-stage training process to maintain disentanglement. Additionally, in some cases, unique motion patterns (like a specific gait) can become entangled with identity. Future work will explore more complex priors for dynamics to handle increasingly unpredictable video sequences.

Summary updated

Abstract

Disentangled representations support a range of downstream tasks including causal reasoning, generative modeling, and fair machine learning. Unfortunately, disentanglement has been shown to be impossible without the incorporation of supervision or inductive bias. Given that supervision is often expensive or infeasible to acquire, we choose to incorporate structural inductive bias and present an unsupervised, deep State-Space-Model for Video Disentanglement (VDSM). The model disentangles latent time-varying and dynamic factors via the incorporation of hierarchical structure with a dynamic prior and a Mixture of Experts decoder. VDSM learns separate disentangled representations for the identity of the object or person in the video, and for the action being performed. We evaluate VDSM across a range of qualitative and quantitative tasks including identity and dynamics transfer, sequence generation, Fr\'echet Inception Distance, and factor classification. VDSM provides state-of-the-art performance and exceeds adversarial methods, even when the methods use additional supervision.
BibTeX
@inproceedings{vowels2021vdsm,
  author = {Matthew J. Vowels and Necati Cihan Camgöz and Richard Bowden},
  title = {{VDSM: Unsupervised Video Disentanglement With State-Space Modeling and Deep Mixtures of Experts}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  doi = {10.1109/cvpr46437.2021.00808},
}
Figure