Unsupervised Learning of Disentangled Representations from Video

Emily Denton, Vighnesh Birodkar

Abstract

We present a new model, DRNET, that learns disentangled image representations from video. Our approach leverages the temporal coherence of video and a novel adversarial loss to learn a representation that factorizes each frame into a stationary part and a temporally varying component. The disentangled representation can be used for a range of tasks. For example, applying a standard LSTM to the time-varying components enables prediction of future frames. We evaluate our approach on a range of synthetic and real videos, demonstrating the ability to coherently generate hundreds of steps into the future.

Introduction

Learning useful representations from video is a fundamental challenge in computer vision. A single video frame contains a complex entanglement of information: object identity, spatial orientation, background context, and lighting. Standard unsupervised models, such as basic auto-encoders, often fail to separate these factors, which leads to "blurry" or "drifting" results in downstream tasks like future frame prediction.

This paper introduces DRNET, a framework designed to achieve disentanglement—separating the "what" (stationary content) from the "where/how" (dynamic pose) without human labels. By factorizing the latent space, we can manipulate specific aspects of a scene and predict future dynamics with significantly higher fidelity than previous state-of-the-art methods.

Methodology

The core of DRNET is a dual-encoder architecture that processes video sequences to extract two distinct latent vectors. The Content Encoder ($E_c$) is designed to capture features that remain constant throughout a video clip, such as object identity. In contrast, the Pose Encoder ($E_p$) extracts features that capture the dynamic, changing aspects of the scene.

Overview of the DRNET training architecture, highlighting the dual-encoder setup and adversarial loss.

To ensure true factorization, we introduce a Scene Discriminator ($C$) and a novel adversarial loss. The discriminator is trained to identify whether two pose vectors originated from the same video sequence. Simultaneously, the Pose Encoder is trained to maximize the uncertainty of this discriminator. This "identity-stripping" mechanism forces the pose vector to be generic regarding object identity, ensuring it only contains information about movement and orientation. Additionally, a similarity loss is applied to the content vectors of adjacent frames to reinforce their stationary nature.

Video Prediction and Latent Space Dynamics

Once the representation is disentangled, predicting future frames becomes a task of modeling the dynamics in the low-dimensional pose space. Instead of predicting complex pixels directly, we use a standard LSTM to predict the next pose vector based on previous frames. Because the pose vector is cleaned of identity information, the LSTM can focus solely on the temporal evolution of the scene. The final predicted frame is generated by passing the static content vector and the predicted pose vector through a frame decoder ($D$).

Experimental Results

DRNET was evaluated on diverse datasets, including Bouncing MNIST, NORB (toy objects), SUNCG (rendered chairs), and the KTH Action Dataset (human actions).

Long-Range Generation

The model demonstrates remarkable stability in long-term video synthesis. On the Bouncing MNIST dataset, DRNET generated crisp, clear frames up to 500 steps into the future. On the real-world KTH dataset, it maintained coherence for 100 steps, significantly outperforming existing models like MCNet, which typically degrade after far fewer frames.

Disentanglement Quality

The quality of the factorization is best illustrated through "content swapping." By combining the content vector from one sequence with the pose vectors from another, we can re-render an object performing a different action.

Visual demonstration of content and pose disentanglement through cross-sequence swapping on Bouncing MNIST.

As shown in the results, the model successfully preserves the identity of the digits (content) while accurately following the motion paths (pose) of the target sequences. This disentanglement also provides a boost to downstream classification tasks; content features proved excellent for object identification, while pose features were superior for action classification.

Conclusion and Future Work

DRNET provides a robust, unsupervised framework for learning factorized representations from video. By isolating stationary content from dynamic pose, the model enables high-quality, long-range video prediction and flexible content manipulation. Future research may address current limitations, such as handling dynamic backgrounds or incorporating probabilistic elements to account for the inherent uncertainty in future motion.

Summary updated

Abstract

We present a new model DrNET that learns disentangled image representations from video. Our approach leverages the temporal coherence of video and a novel adversarial loss to learn a representation that factorizes each frame into a stationary part and a temporally varying component. The disentangled representation can be used for a range of tasks. For example, applying a standard LSTM to the time-vary components enables prediction of future frames. We evaluate our approach on a range of synthetic and real videos, demonstrating the ability to coherently generate hundreds of steps into the future.
BibTeX
@inproceedings{denton2017unsupervised,
  author = {Emily Denton and Vighnesh Birodkar},
  title = {{Unsupervised Learning of Disentangled Representations from Video}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2017},
  url = {https://openreview.net/forum?id=rkbvpD-dZr},
}
Figure