Abstract
We construct an unsupervised learning model that achieves nonlinear disentanglement of underlying factors of variation in naturalistic videos. Previous work suggests that representations can be disentangled if all but a few factors in the environment stay constant at any point in time. As a result, algorithms proposed for this problem have only been tested on carefully constructed datasets with this exact property, leaving it unclear whether they will transfer to natural scenes. Here we provide evidence that objects in segmented natural movies undergo transitions that are typically small in magnitude with occasional large jumps, which is characteristic of a temporally sparse distribution. We leverage this finding and present SlowVAE, a model for unsupervised representation learning that uses a sparse prior on temporally adjacent observations to disentangle generative factors without any assumptions on the number of changing factors. We provide a proof of identifiability and show that the model reliably learns disentangled representations on several established benchmark datasets, often surpassing the current state-of-the-art. We additionally demonstrate transferability towards video datasets with natural dynamics, Natural Sprites and KITTI Masks, which we contribute as benchmarks for guiding disentanglement research towards more natural data domains.
Introduction
In representation learning, "disentanglement" refers to the discovery of independent, interpretable factors of variation—such as object position, color, or rotation—from raw, high-dimensional data. While highly desirable, achieving disentanglement in a purely unsupervised setting faces a fundamental theoretical hurdle: non-identifiability. Without specific constraints, a standard model has no inherent reason to prefer the "true" generative factors over an entangled transformation of them.
Recent research has suggested that temporal structure might be the key to overcoming this. However, most existing methods rely on the assumption that only one or a few factors change between consecutive frames—a condition often met in toy datasets but rarely in the real world. This paper explores whether the inherent temporal statistics of natural data can provide a sufficient mathematical structure for identifiable disentanglement.
Empirical Evidence from Natural Videos
To ground our approach in reality, we analyzed over 300,000 object transitions in natural video datasets, specifically YouTube-VOS and KITTI. Our analysis revealed that transitions in generative factors like position and area are highly sparse. Most of the time, these factors change very little, but they occasionally undergo large "jumps."
This characteristic—a sharp peak at zero with heavy tails—is well-modeled by a generalized Laplace distribution. This empirical finding suggests that natural dynamics are "temporally sparse," providing a realistic basis for disentanglement algorithms.
The SlowVAE Framework and Identifiability
We propose SlowVAE, a Variational Autoencoder designed to leverage these temporal statistics. Unlike standard VAEs that treat data points as independent, SlowVAE models pairs of adjacent frames $(x_{t-1}, x_t)$.
Temporal Sparse Prior
SlowVAE assumes that the transition between latent states $(z_{t-1}, z_t)$ follows a sparse prior (e.g., a Laplace distribution). This contrasts with traditional "slow feature analysis," which uses a Gaussian prior to encourage smooth, small changes. While a Gaussian prior is rotationally invariant—meaning the model cannot distinguish between true axes and rotated versions—the "pointy" nature of a sparse distribution breaks this symmetry.
Identifiability Proof
We provide a mathematical proof showing that if transitions are sparse (non-Gaussian), the true generative factors can be recovered up to simple permutations and sign flips. The sparse prior acts as a natural coordinate system; because the model is penalized for changing too many things at once, it is forced to align its latent axes with the independent factors of the world.
Experimental Setup and Results
We evaluated SlowVAE on standard benchmarks (dSprites, Shapes3D) and two new datasets designed to bridge the gap to natural data: Natural Sprites, with dynamics sampled from real YouTube-VOS transitions, and KITTI Masks, featuring real segmentation masks of pedestrians.
SlowVAE consistently outperformed the previous state-of-the-art, such as Ada-GVAE, across multiple metrics including the Mean Correlation Coefficient (MCC) and Disentanglement, Completeness, and Informativeness (DCI). Notably, on the KITTI Masks dataset, SlowVAE achieved an MCC of 79.6, significantly higher than Ada-GVAE (67.6). Qualitative results from latent traversals further demonstrate that SlowVAE successfully isolates factors like pedestrian scale and position into independent, interpretable dimensions.
Conclusion and Future Work
SlowVAE demonstrates that the temporal sparsity inherent in natural videos is a powerful signal for unsupervised disentanglement. By moving beyond the "one-factor-at-a-time" assumption, we show that identifiable representations can be learned from data with complex, natural dynamics. Future work will focus on relaxing the assumption of factor independence and scaling these methods to full, unsegmented natural video.
Summary updated