Abstract
The identifiability of latent variable models has received increasing attention due to its relevance in interpretability and out-of-distribution generalisation. In this work, we study the identifiability of Switching Dynamical Systems, taking an initial step toward extending identifiability analysis to sequential latent variable models. We first prove the identifiability of Markov Switching Models, which commonly serve as the prior distribution for the continuous latent variables in Switching Dynamical Systems. We present identification conditions for first-order Markov dependency structures, whose transition distribution is parametrised via non-linear Gaussians. We then establish the identifiability of the latent variables and non-linear mappings in Switching Dynamical Systems up to affine transformations, by leveraging identifiability analysis techniques from identifiable deep latent variable models. We finally develop estimation algorithms for identifiable Switching Dynamical Systems. Throughout empirical studies, we demonstrate the practicality of identifiable Switching Dynamical Systems for segmenting high-dimensional time series such as videos, and showcase the use of identifiable Markov Switching Models for regime-dependent causal discovery in climate data.
Introduction
In machine learning, latent variable models (LVMs) are essential for uncovering hidden patterns in complex data. However, these models often suffer from a lack of identifiability: multiple distinct sets of internal parameters can produce the exact same observed data distribution. This ambiguity makes the models "black boxes," where hidden states may not correspond to physical reality.
This paper addresses this challenge for Switching Dynamical Systems (SDSs), a powerful class of models that allow for regime-switching behaviors (e.g., a physical system behaving differently in summer vs. winter). Unlike previous work in non-linear ICA, this framework does not require latent variables to be independent, making it more suitable for interconnected real-world systems.
The SDS and MSM Framework
The authors define the system through two nested models. The Markov Switching Model (MSM) serves as the prior, governing the hidden dynamical part of the system. The broader SDS includes an "emission" model that maps these hidden states to observations.
As shown in the graphical model above, the system involves discrete states $s_t$, continuous latent variables $z_t$, and observations $x_t$. The transition distribution is parameterized via non-linear Gaussians, allowing for complex, regime-dependent dynamics.
Theoretical Foundations
The theoretical contribution of this work is two-fold:
- MSM Identifiability: The authors prove that the prior dynamics are identifiable by treating the MSM as a finite mixture of trajectories. By restricting neural networks to use analytic activation functions (such as SoftPlus or Cosine), they leverage the property that two different analytic functions can only intersect at a negligible number of points. This ensures the model can uniquely distinguish between different switching regimes.
- SDS Identifiability: The proof is extended to the full system, showing that latent variables and non-linear mappings can be recovered up to affine transformations. Crucially, this holds under "weak injectivity," meaning the decoder does not need to be perfectly one-to-one.
Key Results
The researchers validated their theory using synthetic data, climate records, and high-dimensional video data.
Synthetic Verification
In synthetic tests, the model achieved Mean Coefficient Correlation (MCC) scores of ~0.99, indicating near-perfect recovery of true hidden states. As shown in the figures above, the $L_2$ distance between true and estimated functions decreases with sequence length, while $F_1$ scores for state segmentation remain consistently high across various dimensions.
Real-World Applications
- Climate Causal Discovery: On the ENSO-AIR dataset, the model successfully identified that El Niño (ENSO) directly impacts Indian rainfall (AIR) specifically during summer months, aligning with established climate science.
- Video Segmentation: The model was applied to complex dance videos (Salsa and Hip-Hop). It reconstructed these high-dimensional sequences with very low error (Pixel MSE of $2.26 \cdot 10^{-4}$) and provided clean, interpretable segmentation of moves (e.g., "spinning" vs. "standing") without prior labels.
Conclusion and Limitations
This work provides the first theoretical proof of identifiability for SDSs with non-linear transitions and emissions. While a significant step forward, the authors note that current training methods like Variational Inference (ELBO) are still prone to "state collapse" and require careful tuning. Future work will look to extend these proofs beyond first-order Markov chains to systems with longer-term memory.
Summary updated