Abstract
VDSM (Video Disentanglement via State-Space Modeling) is a fully unsupervised framework designed to decompose video sequences into three independent components: identity, pose, and dynamics. By utilizing structural inductive biases, such as a Mixture of Experts (MoE) decoder and hierarchical latent variables, VDSM achieves state-of-the-art performance in video generation and action transfer tasks without the need for manual labels or adversarial training.
Introduction
In the field of computer vision, disentanglement refers to the learning of representations where distinct physical factors—such as a subject's identity versus their specific expression or movement—are isolated into independent variables. This capability is essential for applications like action transfer, where the motion of one subject is mapped onto the appearance of another.
Achieving disentanglement in a purely unsupervised manner is theoretically challenging and typically requires specific "inductive biases" within the model architecture. While existing methods often rely on expensive human-labeled data or unstable Generative Adversarial Networks (GANs), VDSM proposes a solution that leverages the inherent structure of video data to achieve clean disentanglement, avoiding the complexities of supervision and GAN training.
Methodology
VDSM is built upon a Variational Autoencoder (VAE) backbone, enhanced with several structural innovations:
Factorized Latent Space
The model partitions the latent space into three distinct factors:
- Static Factor ($s$): Captures identity and content that remains constant throughout the video.
- Pose Factor ($z_t$): Represents time-varying appearance at each specific frame.
- Dynamics Factor ($d$): A global vector summarizing the overall motion or action sequence.
Mixture of Experts (MoE) Decoder
To prevent identity information from leaking into motion variables, VDSM employs a bank of smaller decoders. The model learns to blend these "experts" based on the identity factor ($s$), forcing the network to specialize its decoding process for specific identities.
Hierarchical State-Space Model
A transition network predicts the subsequent pose ($z_{t+1}$) conditioned on the current pose and the global dynamics factor ($d$), effectively modeling the temporal flow of the video.
Two-Stage Training
Training is conducted in two phases to prevent the model from finding "shortcuts" that bypass disentanglement:
- Stage 1: Focuses on separating identity from pose using shuffled, non-sequential image groups. Temporal averaging is used to ensure the identity factor remains constant.
- Stage 2: Learns sequential dynamics while the identity and pose encoders remain frozen.
Experimental Setup
The performance of VDSM was evaluated across four diverse benchmarks: the MUG Facial Expression Database, Colored Sprites, Moving MNIST, and Synthetic Swinging Pendula. These datasets test the model's ability to handle human faces, animated characters, and physical motion.
Results and Discussion
VDSM demonstrates significant improvements over previous state-of-the-art methods like S3VAE and DSA.
Quantitative Performance
On the MUG dataset, VDSM achieved a Fréchet Inception Distance (FID) of 17.64, significantly outperforming the GAN-based G3AN (67.12), indicating superior visual realism. In action transfer tasks, VDSM improved accuracy consistency by over 30% compared to its closest competitors.
Qualitative Insights
The model successfully performs complex tasks such as swapping facial expressions between individuals and transferring animations (e.g., walking or slashing) between different cartoon sprites with minimal visual artifacts.
Key Insights
- Structure over Supervision: By architectural design—such as forcing the identity factor to be a temporal average—the model is compelled to segregate constant features from dynamic ones.
- Stability: Utilizing a VAE-based approach avoids the common pitfalls of GAN training, such as mode collapse, leading to a more robust and easier-to-train model.
Conclusion
While VDSM shows remarkable success, it currently requires a two-stage training process to maintain disentanglement. Additionally, in some cases, unique motion patterns (like a specific gait) can become entangled with identity. Future work will explore more complex priors for dynamics to handle increasingly unpredictable video sequences.
Summary updated