Causal Future Prediction in a Minkowski Space-Time

Athanasios Vlontzos, Henrique Bergallo Rocha, Daniel Rueckert, Bernhard Kainz

Abstract

Predicting future states in video sequences requires an understanding of physical causality. Standard generative models often fail this by producing physically impossible transitions in flat Euclidean latent spaces. This paper introduces a framework that embeds video frames into a high-dimensional Minkowski space-time, leveraging the geometric properties of light cones to enforce causal consistency. By constraining future predictions to the intersection of light cones from past observations, the model ensures that generated sequences remain physically plausible. Experiments on Moving MNIST and KTH Action Recognition datasets demonstrate improved morphological consistency and identity preservation over time.

Introduction

Predicting the future is a fundamental challenge in computer vision, primarily because machine learning models often lack an inherent understanding of physics. In a standard flat Euclidean latent space, a model might generate a future frame where an object teleports or changes identity because there are no structural rules preventing such transitions. While humans intuitively understand that an event at time $t$ can only affect things within a certain range at time $t+1$, standard neural networks must learn this causality entirely from data.

The authors argue that Euclidean space is fundamentally unsuitable for causal analysis. Instead, they look to the Alexandrov-Zeeman theorem, which suggests that causality requires a Lorentzian group space—the same kind of space-time described by special relativity. This paper proposes treating the latent space of a generative model as a Minkowski space-time to bake causality into the model's topology.

Methodology

The core innovation is the mapping of images into a pseudo-Riemannian manifold where one dimension represents time and the others represent spatial features. This is implemented by extending a Poincaré Variational Autoencoder (VAE) and using a specific diffeomorphism to map the hyperbolic Poincaré ball into Minkowski space.

The Light Cone Constraint

In special relativity, a "light cone" defines the region of space-time that can be reached from a single point given a maximum speed (the speed of light). The authors apply this to latent variables: any causally related future state must lie within the light cone of the current state. This implicitly enforces a maximum rate of change, preventing the "teleportation" or erratic morphing often seen in standard video prediction models.

Causal Intersection

To predict a future frame based on a sequence of past frames, the model calculates the intersection of the light cones for all previous frames. This "causal intersection" significantly narrows down the possible future states, ensuring the prediction is consistent with the entire observed history.

Visualization of the intersecting cones algorithm. The yellow subspace represents the causal intersection of points related to all previous frames.

Experimental Results

The framework was evaluated on two primary datasets: Moving MNIST, featuring digits moving in random continuous translations, and the KTH Action Recognition dataset, which contains real-world videos of human actions.

Morphological Consistency and Identity Preservation

In Experiment 1, the authors demonstrated that as the time allowed for a prediction increases, the variance in the digit's position and shape increases (the cone widens), but the identity of the digit remains stable. This is a direct result of the geometric constraints. Without these constraints, models tend to lose the identity of the object over long-term predictions.

Samples from Experiment 1 showing how the light cone constraint preserves digit identity over time compared to an unconstrained model (bottom row).

Real-World Performance

On the KTH dataset, the model maintained consistent features, such as clothing color, while producing plausible movement. Notably, the model achieved this even though it was only trained on single frames rather than full sequences, highlighting the power of the geometric inductive bias.

Discussion and Future Work

While the results are promising, the current implementation assumes a fixed 45-degree angle for the light cones. Real-world systems exhibit varying levels of "entropy" or uncertainty; for instance, a ball on a track is more constrained than a leaf in the wind. Future work could involve learning the cone aperture (representing thermodynamic entropy) directly from data.

Additionally, the quality of generated images remains dependent on the underlying VAE architecture. Integrating this Minkowski framework into more powerful generative models, such as Diffusion models or GANs, could yield significantly sharper and more complex causal predictions.

Summary updated

Abstract

Estimating future events is a difficult task. Unlike humans, machine learning ap- proaches are not regularized by a natural understanding of physics. In the wild, a plausible succession of events is governed by the rules of causality, which can- not easily be derived from a finite training set. In this paper we propose a novel theoretical framework to perform causal future prediction by embedding spatio- temporal information on a Minkowski space-time. We utilize the concept of a light cone from special relativity to restrict and traverse the latent space of an arbitrary model. We demonstrate successful applications in causal image synthe- sis and future video frame prediction on a dataset of images. Our framework is architecture- and task-independent and comes with strong theoretical guarantees of causal capabilities.
BibTeX
@misc{vlontzos2020causal,
  author = {Athanasios Vlontzos and Henrique Bergallo Rocha and Daniel Rueckert and Bernhard Kainz},
  title = {{Causal Future Prediction in a Minkowski Space-Time}},
  year = {2020},
  eprint = {2008.09154},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2008.09154},
}
Figure