Disentangling Dynamical Systems: Causal Representation Learning Meets Local Sparse Attention

Markus W. Baumgartner, Anson Lei, Joe Watson, Ingmar Posner

TL;DR

This paper proves that the underlying physical parameters of a dynamical system (e.g., masses, spring constants, damping) can be uniquely disentangled from raw trajectory data without predefined equations by enforcing mechanism sparsity. Crucially, the authors prove that considering local, state-dependent causal graphs (such as contact forces that only activate during collisions) strictly expands the conditions under which parameters are identifiable, and they operationalize this using a sparsity-regularized transformer.

What Problem Does This Paper Address?

Traditional parametric system identification methods (like SINDy) discover governing physical equations, but they rely heavily on domain knowledge to specify a library of candidate functions. Conversely, deep learning can model complex physics from data alone, but it operates as a black box: its latent representations are typically entangled and offer no direct insight into the system's underlying physical constants.

The challenge is bridging this divide: Can a model isolate and learn the true physical parameters of a dynamical system directly from trajectory observations without knowing the functional equations beforehand?

Methods

The authors frame system identification as a Causal Representation Learning (CRL) problem, extending the principle of mechanism sparsity (the idea that physical parameters only influence a sparse subset of variables):

  1. Theoretical Identifiability Theorem & Graphical Criterion: The authors establish that a learned latent representation $\hat{\theta}$ recovers the true parameters $\theta$ up to permutation and element-wise diffeomorphism (smooth, invertible non-linear mapping) if:

  2. The state space is path-connected (trajectories can link the domain, ensuring the parameter map does not depend on state).

  3. The learned causal graph is at least as sparse as the true graph ($\|\hat{\mathcal{G}}\|_0 \le \|\mathcal{G}\|_0$).
  4. The transition Jacobian exhibits sufficient variability.
  5. The graph $\mathcal{G}$ satisfies a graphical criterion: $\forall i: \bigcap_{a \in \text{Ch}_{\mathcal{G}}(i)} \text{Pa}_{\mathcal{G}}(a) = \{i\}$.
  6. Broadening Guarantees with State-Dependent Local Graphs: In many physical systems, interactions are intermittent (e.g., two balls only interact during impact). A global causal graph considers every interaction that could ever happen and is often too dense to guarantee identifiability. The authors prove that intersecting across $b$ local causal graphs $\mathcal{G}_l$ yields a strictly tighter graphical criterion: $$\forall i: \bigcap_{l=1}^b \bigcap_{a \in \text{Ch}_{\mathcal{G}_l}(i)} \text{Pa}_{\mathcal{G}_l}(a) = \{i\}$$

  7. Architecture (SPARTAN Decoder in a Conditional VAE):

  8. Encoder: Encodes observed trajectories into latent parameters $\hat{\theta}$.
  9. Decoder: A conditional transformer using SPARTAN, which samples Bernoulli-distributed adjacency masks from attention logits to trace token-to-token paths $\bar{A}(x_0, \hat{\theta})$.
  10. Optimization: Formulated as constrained optimization via dual ascent—minimizing the path-sparsity loss $\mathcal{L}_{\text{path}}$ subject to reconstruction thresholds, combined with an attention logit regularizer $\mathcal{L}_{\text{logit}}$ to prevent softmax saturation and vanishing gradients during training.

Datasets

The authors evaluate their approach on four synthetic dynamical systems with ground-truth parameters:

  • Dual Particle: Two point masses with independent horizontal/vertical damping, where one is attached to a spring; augmented with a counterfactual task. Satisfies the global graphical criterion.
  • Springs: Four point particles interconnected by six candidate springs with random stiffness; modeled with object-level tokens. Satisfies the global graphical criterion.
  • Local Particle: A single particle under damping where the spring becomes slack inside a radius around the origin. Satisfies only the local graphical criterion.
  • Bounce: Five balls with varying masses elastically colliding in a box. Because any ball can collide with any other, its global graph is completely dense, meaning it satisfies only the local graphical criterion.

Key Insights and Ideas

  • Local Sparsity Strictly Dominates Global Sparsity: Global causal summaries erase temporal sparsity. Evaluating causal graphs within local open partitions of the state space restricts the set of admissible latent representations monotonically, making systems identifiable that were previously theoretically unidentifiable.
  • Path Connectedness Prevents Latent Leaks: If system trajectories cannot traverse between different regions of the state space, the encoder can lazily learn different parameter representations for different regions. Path connectedness guarantees the parameter mapping remains state-invariant.
  • Soft Attention Isn't Always Enough: While softmax attention provides an inductive bias towards sparsity, explicit path regularization is necessary when graphs change discontinuously.

Key Results

Performance was evaluated using the Mean-Max Correlation Coefficient (MCC), where 1.0 represents perfect disentanglement up to element-wise diffeomorphism:

  • Global Criterion Settings (Dual Particle and Springs): Both static causal discovery baselines (VCD) and the proposed local sparse transformer (SPARTAN) achieved near-perfect disentanglement ($\text{MCC} \approx 0.95\text{--}1.0$), while standard MLPs ($\text{MCC} \approx 0.5\text{--}0.7$) and unregularized Transformers failed to reliably disentangle.
  • Local Criterion Settings (Local Particle and Bounce):
  • In Local Particle, VCD achieved only partial disentanglement ($\text{MCC} \approx 0.72$), whereas SPARTAN achieved $\text{MCC} \approx 0.95$.
  • In Bounce, VCD failed completely ($\text{MCC} \approx 0.52$) because the global graph is fully connected. SPARTAN attained an $\text{MCC} \approx 0.98$.
  • Latent Interventions: Perturbing individual learned latent dimensions in SPARTAN systematically modulated isolated physical attributes (e.g., selectively altering horizontal damping while leaving spring stiffness untouched).

Limitations and Future Work

  • Known Latent Dimensionality: The current theoretical formulation assumes the true dimension of the parameter space is known beforehand. Automatic latent dimension model selection is left for future work.
  • Synthetic Verification Only: Experiments were confined to simulated 2D physical environments; testing on high-dimensional real-world systems (e.g., climate or robotics data) remains to be demonstrated.
  • Active Data Acquisition: The graphical criterion defines what is identifiable under observed distributions; the authors suggest using this criterion to guide active data exploration to purposefully trigger unobserved local causal modes.

Summary updated

Abstract

Parametric system identification methods estimate the parameters of explicitly defined physical systems from data. Yet, they remain constrained by the need to provide an explicit function space, typically through a predefined library of candidate functions chosen via available domain knowledge. In contrast, deep learning can demonstrably model systems of broad complexity with high fidelity, but black-box function approximation typically fails to yield explicit descriptive or disentangled representations revealing the structure of a system. We develop a novel identifiability theorem, leveraging causal representation learning, to uncover disentangled representations of system parameters without structural assumptions. We derive a graphical criterion specifying when system parameters can be uniquely disentangled from raw trajectory data, up to permutation and diffeomorphism. Crucially, our analysis demonstrates that global causal structures provide a lower bound on the disentanglement guarantees achievable when considering local state-dependent causal structures. We instantiate system parameter identification as a variational inference problem, leveraging a sparsity-regularised transformer to uncover state-dependent causal structures. We empirically validate our approach across four synthetic domains, demonstrating its ability to recover highly disentangled representations that baselines fail to recover. Corroborating our theoretical analysis, our results confirm that enforcing local causal structure is often necessary for full identifiability.
BibTeX
@inproceedings{baumgartner2026disentangling,
  author = {Markus W. Baumgartner and Anson Lei and Joe Watson and Ingmar Posner},
  title = {{Disentangling Dynamical Systems: Causal Representation Learning Meets Local Sparse Attention}},
  booktitle = {CLeaR},
  year = {2026},
  url = {https://openreview.net/forum?id=5jPq695J19},
}

Loading related content...

Figure