Partitioning Latent Spaces into Static and Dynamic Factors

54 concepts43 papers1 proposal

When we observe a sequence of events—whether it is a video of a person walking, a medical time series of a patient, or a physical system like a swinging pendulum—our brains naturally separate the entity from its behavior. We recognize that a pendulum's length is a fixed, unchanging property, while its velocity changes from millisecond to millisecond.

However, standard sequential models like Recurrent Neural Networks (RNNs) or standard Neural Ordinary Differential Equations (NODEs) do not inherently make this distinction. They collapse both time-invariant traits (identity, physical constants) and time-varying states (motion, transient noise) into a single evolving latent vector. This conflation forces the model to continuously re-learn how static traits manifest at every time step, leading to brittle generalizations and an inability to handle novel combinations of identity and motion.

To overcome this limitation, a powerful paradigm has emerged: static-dynamic latent factorization. By architecturally partitioning the latent space into independent time-invariant and time-varying components, models can achieve superior generalization, robust forecasting, and the ability to perform counterfactual reasoning.

The Architectural Blueprint: Global vs. Local Latents

The fundamental mechanism behind sequential disentanglement is the structural partitioning of the latent space. Instead of using a single bottleneck vector per time step, the architecture is split into two distinct pathways: a single global latent variable that remains constant across the entire sequence, and a sequence of local latent variables that evolve over time.

Standard Sequential VAE:
[Frame 1] ---> [ Latent z1 ] ---> Reconstructed Frame 1
[Frame 2] ---> [ Latent z2 ] ---> Reconstructed Frame 2

Factorized Sequential VAE:
[Sequence] -> [ Global Latent (Static/Identity) ] ---\
                                                      +---> Reconstructed Frames
[Frame t]  -> [ Local Latent t (Dynamic/Motion) ] ---/

In Disentangled Sequential Autoencoder, Yingzhen Li et al. demonstrated that this simple structural constraint forces a Variational Autoencoder (VAE) to isolate time-invariant attributes (like a speaker's identity) from time-varying dynamics (like the specific words spoken). Because the global variable has no temporal index, it cannot capture motion without incurring a massive reconstruction penalty across the rest of the sequence. Conversely, because the local variables are forced to explain frame-to-frame variations, they are disincentivized from storing static traits that are already efficiently compressed by the global channel.

This architectural blueprint has been successfully adapted across diverse domains. For instance, Martin Renqiang Min et al. extended this to self-supervised video generation in S3VAE: Self-Supervised Sequential VAE for Representation Disentanglement and Data Generation by leveraging temporal shuffling and optical flow as free supervisory signals. Similarly, in graph-structured data, Junruo Gao et al. utilized a factorized VAE in Helper Recommendation with seniority control in Online Health Community to separate a user's intrinsic, stable characteristics from their evolving, stage-dependent states.

Modulating Continuous Physical Dynamics

The problem of conflating static and dynamic features becomes even more pronounced in continuous-time modeling, such as physics simulations. Standard Neural ODEs parameterize an entire trajectory based solely on an initial state $h_0$. If two systems follow the same underlying physical laws but possess different static parameters—such as two pendulums with different lengths or friction coefficients—a standard Neural ODE is forced to map these differences into entirely separate trajectory spaces.

To address this, Ilze Amanda Auzina et al. introduced Modulated Neural ODEs, which augment the latent space with explicit time-invariant "modulator" variables. Instead of forcing the dynamic state to carry the burden of static identity, the modulator acts as a fixed parameter that conditions the derivative function itself:

$$\frac{dh(t)}{dt} = f(h(t), \mathbf{m})$$

where $\mathbf{m}$ is the static modulator and $h(t)$ is the evolving dynamic state. This prevents the model from confusing the phase of motion with the underlying physical parameters.

But why does this separation remain stable over long prediction horizons? Jérémie DONA et al. provided a mathematical foundation for this in PDE-Driven Spatiotemporal Disentanglement by leveraging the classical "separation of variables" method from Partial Differential Equations (PDEs). They proved that because the flow of a deterministic differential equation is invertible, if the spatial (static) and temporal (dynamic) representations are successfully partitioned at the initial state, the mathematical structure of the system guarantees that the information remains strictly partitioned as the system evolves.

This continuous-time factorization has been further refined by Deqian Kong et al. in Latent Space Energy-based Neural ODEs, replacing simplistic Gaussian priors with learnable Energy-Based Models (EBMs) to better capture complex, irregularly sampled sequences without losing track of the invariant physical coefficients.

Enforcing Independence Beyond Structure

While architectural partitioning provides a strong inductive bias, structure alone cannot completely prevent information leakage. For example, a model might still bleed static information into the dynamic channel to minimize short-term reconstruction errors. To achieve true disentanglement, researchers introduce explicit mathematical constraints to enforce independence between the two spaces.

Mutual Information and Orthogonality Constraints

One direct approach is to penalize the statistical dependence between the static and dynamic latents. In Decoupling Local and Global Representations of Time Series, Sana Tonekaboni et al. introduced a counterfactual regularization technique designed to minimize the mutual information between local and global latent spaces, ensuring that local states only capture transient variations.

In domains like relational reasoning, Qian Chen et al. enforced strict mathematical orthogonality between representations in Beyond Entity Correlations: Disentangling Event Causal Puzzles in Temporal Knowledge Graphs. By forcing the vector dot product of time-invariant semantics and time-dependent reactions to be zero, the model successfully isolates long-term stable factors from short-term shocks.

Contrastive and Generative Objectives

Instead of explicit penalty terms, disentanglement can also be driven through contrastive learning. Ilan Naiman developed a modality-free framework called Sample and Predict Your Latent: Modality-free Sequential Disentanglement via Contrastive Estimation (SPYL). SPYL samples positive and negative "views" directly from the internal distributions of a VAE's latent space, maximizing mutual information across sequences that share the same static factor but possess different dynamics.

Recently, this concept has been elevated to diffusion models. Hedi Zisling et al. introduced DiffSDA: Unsupervised Diffusion Sequential Disentanglement Across Modalities, which replaces complex, multi-loss VAE and GAN objectives with a single, unified score-estimation loss. By using recurrent architectures to summarize temporal information into distinct static and dynamic diffusion trajectories, it achieves state-of-the-art sequential disentanglement across video, audio, and medical time-series data.

Why It Works: Downstream Advantages

Decoupling the static "what" from the dynamic "how" yields significant practical advantages over traditional, entangled sequential models.

  • Counterfactual Generation and Content Swapping: Because the static and dynamic factors are mathematically independent, they can be recombined arbitrarily. In Disentangled Counterfactual Learning for Physical Audiovisual Commonsense Reasoning, Changsheng Lv et al. isolated static material properties from dynamic interactions, allowing an AI to simulate what an event would sound or look like if an object's material were changed (e.g., swapping glass for wood). In clinical settings, Jong-Hoon Ahn et al. leveraged a constant latent state in A Linear Algebraic Framework for Counterfactual Generation to project a patient's fixed baseline history through time-varying treatment matrices, accurately predicting individual treatment effects.
  • Noise Robustness and Change Point Detection: In time-series analysis, high-frequency noise or phase shifts frequently trigger false alarms in anomaly detection. By explicitly separating instantaneous variations from the stable underlying process, the framework developed by Tim De Ryck et al. in Change Point Detection in Time Series Data using Autoencoders with a Time-Invariant Representation allows change-point detection algorithms to focus exclusively on structural shifts in the system rather than transient environmental noise.
  • High-Level Intention Inference: When analyzing human behavior, momentary adjustments or physical obstacles can obscure an agent's overarching goal. As shown by Mark Zolotas et al. in Disentangled Sequence Clustering for Human Intention Inference, clustering sequences exclusively on their time-invariant global features allows a model to infer high-level human intentions (such as navigating to a specific room) without being distracted by the low-level noise of momentary physical adjustments.

Key Foundational Work

Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data (2017) — contributed 1 concept elements to this direction.
Reconstruction of Sequential Data with Probabilistic Models and Continuity Constraints (1999) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_spatiotemporal_feature_disentanglement_266c2698,
  title = {spatiotemporal feature disentanglement},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/spatiotemporal-feature-disentanglement/rdr_266c26982f3a4ae3bdfd30f36248d14a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_spatiotemporal_feature_disentanglement_266c2698_v1,
  title = {spatiotemporal feature disentanglement},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/spatiotemporal-feature-disentanglement/rdr_266c26982f3a4ae3bdfd30f36248d14a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake