Probing the relationship between linear dynamical systems and low-rank recurrent neural network models

Adrian Valente, Srdjan Ostojic, Jonathan W. PillowarXiv 2021 · DOI · arXiv

Abstract

A large body of work has suggested that neural populations exhibit low-dimensional dynamics during behavior. However, there are a variety of different approaches for modeling low-dimensional neural population activity. One approach involves latent linear dynamical system (LDS) models, in which population activity is described by a projection of low-dimensional latent variables with linear dynamics. A second approach involves low-rank recurrent neural networks (RNNs), in which population activity arises directly from a low-dimensional projection of past activity. Although these two modeling approaches have strong similarities, they arise in different contexts and tend to have different domains of application. Here we examine the precise relationship between latent LDS models and linear low-rank RNNs. When can one model class be converted to the other, and vice versa? We show that latent LDS models can only be converted to RNNs in specific limit cases, due to the non-Markovian property of latent LDS models. Conversely, we show that linear RNNs can be mapped onto LDS models, with latent dimensionality at most twice the rank of the RNN.

What Problem Does This Paper Address?

In systems neuroscience, researchers often observe that the activity of thousands of neurons is driven by just a few "latent" variables. To model this, two main frameworks have emerged:

  1. Descriptive Statistical Models (LDS): These treat neural activity as a noisy projection of hidden variables that evolve linearly over time. They are widely used for statistical inference, such as using Kalman filters to estimate hidden states.
  2. Mechanistic Models (RNNs): These model activity as a direct result of neurons interacting through a connectivity matrix. Low-rank RNNs are a popular subset that naturally produces low-dimensional dynamics.

Despite their similarities, these models come from different traditions—engineering versus physics and biology—and possess distinct mathematical properties. This paper asks: Are they equivalent? Can a model trained as an LDS be translated into an RNN, and vice versa?

Comparison of latent Linear Dynamical Systems (LDS) and low-rank Recurrent Neural Networks (RNNs) architectures and equations.

Methods

The authors perform a rigorous mathematical comparison of the two model classes by examining the joint probability distributions of the time series they generate. Key analytical tools include:

  • Kalman Filter Framework: Used to show how an LDS integrates history to estimate its latent state.
  • Subspace Mapping: Identifying the linear subspace spanned by the singular vectors of the RNN's connectivity matrix to map it to an LDS.
  • Limit Case Analysis: Investigating model behavior as the number of recorded neurons ($n$) becomes much larger than the number of latent dimensions ($d$).

Key Insights and Ideas

The Markovian Gap

The fundamental difference lies in memory. Linear RNNs are Markovian, meaning the next state depends only on the current state. In contrast, the observed output of an LDS is non-Markovian. Because of observation noise, the true latent state is never perfectly known; an observer must integrate the entire history of activity to form an optimal estimate.

The "Large $n$" Bridge

A key finding is that as the number of recorded neurons increases ($n \gg d$), the observation noise effectively averages out. In this limit, current activity provides sufficient information to identify the latent state perfectly, making the LDS effectively Markovian and thus equivalent to an RNN.

Rank vs. Dimensionality

The paper proves that any linear low-rank RNN of rank $r$ can be mapped onto an LDS. However, the latent dimensionality of that LDS may need to be up to $2r$ to account for the potential misalignment between neurons that drive dynamics and those that reflect them.

Key Results

  • LDS to RNN: An LDS can be approximated by a low-rank RNN by matching one-step transitions. This approximation becomes exact when observation noise vanishes or the population size $n \to \infty$.
  • RNN to LDS: A noisy low-rank RNN can always be mapped to an LDS. The latent state of this LDS is defined by the projection of neural activity onto the subspace spanned by the RNN's connectivity components.
  • Convergence: Synthetic simulations demonstrate that for small populations, the autocovariance of an LDS and its RNN approximation differ significantly. However, as the population size grows (e.g., $n=100$), the models become virtually indistinguishable.

Limitations and Future Work

The current work focuses on linear systems with additive Gaussian noise. While the authors discuss potential extensions to non-linear models (e.g., ReLU or tanh activations), formal proofs for these cases remain for future study. Additionally, real neural data often involves non-Gaussian "spike" noise, which presents an open challenge. Finally, the authors note that even when models are mathematically equivalent, the practical results may differ based on the fitting procedure used (e.g., Expectation-Maximization for LDS vs. Backpropagation for RNNs).

Summary updated

Abstract

A large body of work has suggested that neural populations exhibit low-dimensional dynamics during behavior. However, there are a variety of different approaches for modeling low-dimensional neural population activity. One approach involves latent linear dynamical system (LDS) models, in which population activity is described by a projection of low-dimensional latent variables with linear dynamics. A second approach involves low-rank recurrent neural networks (RNNs), in which population activity arises directly from a low-dimensional projection of past activity. Although these two modeling approaches have strong similarities, they arise in different contexts and tend to have different domains of application. Here we examine the precise relationship between latent LDS models and linear low-rank RNNs. When can one model class be converted to the other, and vice versa? We show that latent LDS models can only be converted to RNNs in specific limit cases, due to the non-Markovian property of latent LDS models. Conversely, we show that linear RNNs can be mapped onto LDS models, with latent dimensionality at most twice the rank of the RNN.
BibTeX
@misc{valente2021probing,
  author = {Adrian Valente and Srdjan Ostojic and Jonathan W. Pillow},
  title = {{Probing the relationship between linear dynamical systems and low-rank recurrent neural network models}},
  year = {2021},
  eprint = {2110.09804},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2110.09804},
}
Figure