Abstract
Predicting future states in video sequences requires an understanding of physical causality. Standard generative models often fail this by producing physically impossible transitions in flat Euclidean latent spaces. This paper introduces a framework that embeds video frames into a high-dimensional Minkowski space-time, leveraging the geometric properties of light cones to enforce causal consistency. By constraining future predictions to the intersection of light cones from past observations, the model ensures that generated sequences remain physically plausible. Experiments on Moving MNIST and KTH Action Recognition datasets demonstrate improved morphological consistency and identity preservation over time.
Introduction
Predicting the future is a fundamental challenge in computer vision, primarily because machine learning models often lack an inherent understanding of physics. In a standard flat Euclidean latent space, a model might generate a future frame where an object teleports or changes identity because there are no structural rules preventing such transitions. While humans intuitively understand that an event at time $t$ can only affect things within a certain range at time $t+1$, standard neural networks must learn this causality entirely from data.
The authors argue that Euclidean space is fundamentally unsuitable for causal analysis. Instead, they look to the Alexandrov-Zeeman theorem, which suggests that causality requires a Lorentzian group space—the same kind of space-time described by special relativity. This paper proposes treating the latent space of a generative model as a Minkowski space-time to bake causality into the model's topology.
Methodology
The core innovation is the mapping of images into a pseudo-Riemannian manifold where one dimension represents time and the others represent spatial features. This is implemented by extending a Poincaré Variational Autoencoder (VAE) and using a specific diffeomorphism to map the hyperbolic Poincaré ball into Minkowski space.
The Light Cone Constraint
In special relativity, a "light cone" defines the region of space-time that can be reached from a single point given a maximum speed (the speed of light). The authors apply this to latent variables: any causally related future state must lie within the light cone of the current state. This implicitly enforces a maximum rate of change, preventing the "teleportation" or erratic morphing often seen in standard video prediction models.
Causal Intersection
To predict a future frame based on a sequence of past frames, the model calculates the intersection of the light cones for all previous frames. This "causal intersection" significantly narrows down the possible future states, ensuring the prediction is consistent with the entire observed history.
Experimental Results
The framework was evaluated on two primary datasets: Moving MNIST, featuring digits moving in random continuous translations, and the KTH Action Recognition dataset, which contains real-world videos of human actions.
Morphological Consistency and Identity Preservation
In Experiment 1, the authors demonstrated that as the time allowed for a prediction increases, the variance in the digit's position and shape increases (the cone widens), but the identity of the digit remains stable. This is a direct result of the geometric constraints. Without these constraints, models tend to lose the identity of the object over long-term predictions.
Real-World Performance
On the KTH dataset, the model maintained consistent features, such as clothing color, while producing plausible movement. Notably, the model achieved this even though it was only trained on single frames rather than full sequences, highlighting the power of the geometric inductive bias.
Discussion and Future Work
While the results are promising, the current implementation assumes a fixed 45-degree angle for the light cones. Real-world systems exhibit varying levels of "entropy" or uncertainty; for instance, a ball on a track is more constrained than a leaf in the wind. Future work could involve learning the cone aperture (representing thermodynamic entropy) directly from data.
Additionally, the quality of generated images remains dependent on the underlying VAE architecture. Integrating this Minkowski framework into more powerful generative models, such as Diffusion models or GANs, could yield significantly sharper and more complex causal predictions.
Summary updated