Tree-Structured Recurrent Switching Linear Dynamical Systems for Multi-Scale Modeling

Josue Nassar, Scott W. Linderman, Mónica F. Bugallo, Il Memming Park

Understanding complex, nonlinear systems—from the firing patterns of neurons to the chaotic flow of the atmosphere—is a fundamental challenge in science. Researchers usually face a frustrating trade-off when modeling these dynamics:

  1. Interpretable but simple models: Linear systems are easy to understand but fail to capture the "twists and turns" of real-world data.
  2. Accurate but "black-box" models: Deep neural networks or Gaussian processes can predict future states with high accuracy but offer little insight into how the system is actually behaving.

Existing "Switching Linear Dynamical Systems" (SLDS) try to bridge this gap by switching between different linear regimes. However, standard variants (like recurrent SLDS) partition space sequentially. This makes them sensitive to the order of states and prevents them from capturing the natural hierarchical structure often found in complex systems.

This paper introduces the Tree-Structured Recurrent Switching Linear Dynamical System (TrSLDS), a probabilistic model that approximates complex nonlinear dynamics using a hierarchy of simpler, locally linear models. By replacing sequential partitioning with a tree structure, the model allows researchers to "zoom in" on data, moving from coarse, interpretable global descriptions to high-fidelity, nuanced local details.

The TrSLDS Approach

The TrSLDS organizes its linear regimes into a hierarchical tree, introducing several key innovations:

Tree-Structured Stick-Breaking: Instead of carving up the state space in a simple sequence, the model uses a binary tree. At each node, the system makes a "left or right" choice based on its current state. This allows the model to partition space more flexibly and logically, naturally avoiding the permutation problems that plague sequential models.

Comparison of sequential stick-breaking versus tree-structured stick-breaking.

Hierarchical Dynamics Prior: This is the model's "secret sauce." It imposes a prior where parent nodes in the tree represent coarse, global dynamics, while child nodes learn the "residuals"—the specific nuances that differentiate one region of space from another. The authors prove that this hierarchical prior is mathematically equivalent to a "residual model" where the final dynamics are a sum of the transformations learned at every level of the tree.

Efficient Inference: To make the model practical, the researchers use Pólya-Gamma data augmentation. This mathematical trick turns non-conjugate logistic regressions into Gaussian forms, allowing for fast, "conjugate" Gibbs sampling. As a result, the model can be fit to large datasets much faster than traditional gradient-based methods for complex hierarchies.

Key Insights and Results

The model was evaluated on both synthetic benchmarks (like the FitzHugh-Nagumo oscillator and Lorenz attractor) and real-world biological data (Macaque V1 neural spike trains).

Multi-Scale Interpretability: The most striking feature of TrSLDS is the ability to query the model at different depths. You can look at the "root" node to see the general flow of the system, then look at the "leaves" to see fine-grained oscillations. For example, on the Lorenz attractor, the second level of the tree successfully captured the two main "wings" of the butterfly, while the leaf nodes captured the specific spiraling nuances within those wings.

Superior Prediction: In both the FHN and Lorenz experiments, the TrSLDS significantly outperformed standard rSLDS and LDS in $R^2$ predictive power, especially as the prediction horizon (number of steps into the future) increased.

Vector fields and k-step prediction performance showing TrSLDS outperforming baseline models.

Real-World Utility: When applied to macaque neural data, the model correctly identified and separated the limit cycles (periodic orbits) corresponding to different visual stimulus orientations, providing a clearer picture of the underlying neural manifold than previous methods.

Limitations and Future Work

While powerful, the TrSLDS model has a few limitations:

  • Identifiability: Like many state-space models, TrSLDS is susceptible to rotational and scaling invariance. The authors implemented specific RQ decomposition steps during sampling to prevent parameters from "spinning" or scaling indefinitely.
  • Initialization Sensitivity: Because the model is complex and hierarchical, it requires a "greedy" initialization strategy to start the Gibbs sampler in a good region of the parameter space.
  • Tree Topology: Currently, the user must specify the tree structure (e.g., a balanced binary tree). Future work could involve "learning" the tree structure itself from the data, potentially using Bayesian non-parametrics.

By framing complex dynamics as a simple global system plus a series of increasingly localized corrections, TrSLDS offers a powerful new lens for understanding the nonlinear world.

Summary updated

Abstract

Many real-world systems studied are governed by complex, nonlinear dynamics. By modeling these dynamics, we can gain insight into how these systems work, make predictions about how they will behave, and develop strategies for controlling them. While there are many methods for modeling nonlinear dynamical systems, existing techniques face a trade off between offering interpretable descriptions and making accurate predictions. Here, we develop a class of models that aims to achieve both simultaneously, smoothly interpolating between simple descriptions and more complex, yet also more accurate models. Our probabilistic model achieves this multi-scale property through of a hierarchy of locally linear dynamics that jointly approximate global nonlinear dynamics. We call it the tree-structured recurrent switching linear dynamical system. To fit this model, we present a fully-Bayesian sampling procedure using Polya-Gamma data augmentation to allow for fast and conjugate Gibbs sampling. Through a variety of synthetic and real examples, we show how these models outperform existing methods in both interpretability and predictive capability.
BibTeX
@inproceedings{nassar2019tree,
  author = {Josue Nassar and Scott W. Linderman and Mónica F. Bugallo and Il Memming Park},
  title = {{Tree-Structured Recurrent Switching Linear Dynamical Systems for Multi-Scale Modeling}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=UIeyVC_lP4P},
}
Figure