Introduction
In standard representation learning, deep models often treat latent variables as "black boxes." While a Variational Auto-Encoder (VAE) might learn to compress a video of a swinging pendulum, the resulting latent numbers do not necessarily correspond to physical variables like angle or velocity in a human-interpretable way. Furthermore, these models struggle with "out-of-distribution" scenarios; they cannot easily imagine how physics would change if a single variable, such as gravity or friction, were altered.
This paper introduces a framework for training VAEs that learn physically meaningful latent variables from video. By replacing standard, uninformative Gaussian priors with nonlinear "Additive Noise Models" (ANMs) based on physical equations, the authors enable models to not only reconstruct video but also perform "counterfactual reasoning"—generating realistic videos of scenarios never seen during training, such as a pendulum swinging with air drag or an object falling under Jupiter's gravity.
Methods
The core innovation lies in the VAE prior, which moves beyond simple isotropic Gaussians to incorporate physical mechanisms.
- Nonlinear Additive Noise Model (ANM): The prior is specified as $y = f(t) + n$, where $f(t)$ is a known physical equation (e.g., a sine wave for an oscillator) and $n$ is Gaussian noise.
- Local Linearization: Since a nonlinear function of a Gaussian variable is non-Gaussian, the authors use a first-order Taylor expansion to approximate the prior as a Gaussian Mixture Model (GMM).
- Stable KL Divergence: To handle the analytically impossible KL divergence between two GMMs, the authors derived a numerically stable, parallelizable Monte Carlo estimate using
logsumexpandlogdetoperations. - Intervention: Once trained, the physical mechanism $f(t)$ can be swapped. For instance, a model trained on earth-like gravity can be given a new equation for Jupiter's gravity, allowing the decoder to generate video for this "imagined" scenario.
Datasets
The method was validated on four datasets involving high-dimensional raw RGB frames governed by simple physical laws:
- Mass on a Spring: Vertical oscillation of a weight.
- Pendulum: A swinging weight on a string.
- Falling Object: Free fall from a height.
- Crab Pulsar: Astronomical data of a rotating neutron star emitting periodic pulses.
Key Results and Insights
The proposed method forces the encoder to find specific physical variables within pixels by using physics as an inductive bias. This approach bridges the gap between deep learning and structural equation modeling.
- Latent Accuracy: The model achieved a Mean Squared Error (MSE) for latent variables 3 to 4 orders of magnitude lower than standard VAE baselines, proving it learned the "true" physical variables.
- Reconstruction: High reconstruction accuracy (97.5% to 99.9%) was maintained despite the stricter prior, comparable to models that use time as a direct input.
- Visual Counterfactuals: The model successfully generated "imagined" videos. For the pendulum, adding a friction term allowed the model to render a slowing swing, even though it was only trained on constant-amplitude data. Similarly, it simulated faster falling speeds under Jupiter’s gravity.
Conclusion
While the current approach requires a known physical equation and focuses on single-variable scenes, it demonstrates a powerful method for achieving identifiability in representation learning. Future work may involve learning the physical mechanisms directly from data or scaling to multi-variate, complex environments.
Summary updated