Shared Dynamics, Local Noise: Count-Likelihood Conditionally Linear Models Pooled Across Sessions

Neuroscientists often want a model of how a brain circuit's activity evolves, a "dynamical system", that they can analyze by hand. They want to find its resting points and see how those points move as behavior changes. One recent model does this well: the conditionally linear dynamical system (CLDS). It is linear at any fixed value of a behavioral variable such as reach angle or head direction, and it lets the linear rules change smoothly as that variable changes. As a result it works from a handful of trials and gives its resting points ("fixed points") in closed form. Its weakness, which its own authors flag, is that it treats recorded activity as Gaussian, meaning continuous, symmetric, and with constant noise. Real spike counts are none of these: they are non-negative integers, skewed at low firing rates, and their variance grows with their mean. A second gap is that the model is fit to one dataset at a time. Most labs, however, hold many short recording sessions, each from a different set of neurons, and possibly from different recording methods.

This note proposes a model that splits along those two gaps. Each session keeps its own local measurement model: which neurons it reads out, their baseline rates, and their own count-noise distribution (Poisson, overdispersed, or zero-inflated). The behavior-dependent dynamics are shared across all sessions and estimated from the pooled data. The central claim is that the noise model and the parameter split are linked. When sessions are pooled, a misspecified Gaussian likelihood in a low-rate or calcium session does more than lower that session's fit. It pulls the shared dynamics and biases the shared fixed points. Moving the noise model into the local, per-session parameters protects the shared part. Count likelihoods can be added without losing the closed-form update for the dynamics, because the non-Gaussian part enters only the observation equation.

Core Claim

The model has three equations. For session $s$ with neurons $n=1..N_s$, latent state $x_t$, and observed behavioral covariate $u_t$, the shared dynamics are

$$x_{t+1} = A(u_t)\,x_t + b(u_t) + \epsilon_t,\qquad \epsilon_t \sim \mathcal{N}(0,Q).$$

The local spike-count observation model is

$$y^{(s)}_{n,t} \sim \mathrm{NB}\big(r^{(s)}_n,\ \sigma(c^{(s)\top}_n x_t + d^{(s)}_n)\big).$$

Here $A(u)$ and $b(u)$ are functions of behavior with Gaussian-process priors, exactly as in Modeling Neural Activity with Conditionally Linear Dynamical Systems. The readout weights $c^{(s)}_n$, offsets $d^{(s)}_n$, and dispersions $r^{(s)}_n$ belong only to session $s$. NB is the negative binomial distribution with logistic link $\sigma$; as $r\to\infty$ it approaches Poisson. For calcium sessions, a zero-inflated variant replaces the negative binomial. The fixed points keep the CLDS form $(I-A(u))x^*=b(u)$, and these are the analytic objects we care about.

The technical point that makes this feasible is as follows. With a standard auxiliary-variable augmentation of the negative-binomial likelihood (Pólya-gamma variables), the observation model becomes conditionally Gaussian once the auxiliary variables are fixed. The latent smoother can then stay an exact Kalman smoother within each Gibbs or variational sweep. The update for the shared $A(u), b(u)$ depends only on expected latent sufficient statistics, so it is the same Sylvester-equation solve that CLDS uses. Those statistics simply add across sessions. The shared update therefore never sees raw spikes or session readouts, only summed second moments of the latent states. That makes the same algorithm usable in a federated setting without changes, although federation is a secondary benefit here rather than the main claim.

Relation to Prior Work

CLDS supplies the method and the motivation. It recovers ring attractors in mouse head-direction thalamus and beats LDS, gpSLDS and LFADS on held-out-neuron prediction in premotor cortex with 1–5 trials per condition. Its authors name the Gaussian observation assumption as the main open limitation (Modeling Neural Activity with Conditionally Linear Dynamical Systems). We keep its dynamics and change its observation model and its single-dataset scope.

The case that the noise model matters comes from several papers:

None of these study what misspecification does to dynamics shared across sessions, and that is the question tested here.

The closest existing system is LFADS with "neural stitching" (Inferring single-trial neural population dynamics using sequential auto-encoders). It already combines a Poisson likelihood, nonlinear recurrent dynamics, a shared generator, and per-session read-in and read-out layers, and stitching raised kinematic $R^2$ by 0.18. Our proposal differs in three ways. Its dynamics are conditionally linear with closed-form fixed points rather than an opaque recurrent network. It targets the regime of few trials per session. And it makes the noise family itself, not only the readout matrix, a local parameter.

Three other cross-session or cross-region models are relevant:

On the decoupling side, Federated neural nonparametric point processes is the closest analogue. It keeps client-specific mean and scale parameters local and shares kernel hyperparameters, so it is evidence that decoupled point-process likelihoods survive sparse clients. It models scalar event streams, however, not population state spaces. Towards Personalized Federated Learning frames the split between global and local parameters that we borrow.

Research Program

We will build the model in three stages and test whether its extra structure is needed at each one.

The first stage is a Poisson and negative-binomial CLDS for a single session. It uses the augmented Kalman E-step and the unchanged Sylvester M-step. We check that it reduces to Gaussian CLDS when firing rates are high.

The second stage adds pooling across sessions. Dynamics are shared and readouts and dispersions are local. Latent coordinates are fixed by anchoring the shared dynamics, while readouts absorb the arbitrary rotation each session would otherwise pick. This is our answer to the representation-drift problem that the decoupling literature raises.

The third stage is a mixed-modality setting. Electrophysiology and calcium sessions share dynamics, and each session uses its own likelihood family.

Alongside these stages runs a controlled synthetic study. We simulate a ring attractor and a two-fixed-point decision system, then generate sessions with varied firing rates, dispersion, and zero inflation, so the true fixed points are known. This study answers the central question directly: does a pooled Gaussian model bias the shared fixed points, and does moving the noise model into local parameters remove that bias?

Baselines

The naive reference point is the per-neuron mean-firing-rate model. On NLB co-smoothing it scores 0 bits per spike by construction, because the metric is defined relative to it. The classical linear reference is GPFA, which Neural Latents Benchmark '21 reports at 0.187 bps co-smoothing on MC_Maze.

The strongest existing method is AutoLFADS with stitching. On NLB MC_Maze it reports 0.346 bps co-smoothing, the highest generative number in the file. We will run it with the published hyperparameter search, and its search budget will be matched in wall-clock time by our own tuning of the latent dimension and GP length scales. On full MC_Maze we do not expect to beat 0.346 bps, given that our dynamics are more restricted. Our model counts as beating AutoLFADS if it matches it within noise on the MC_Maze Small split, and exceeds it when each session is cut to 1–5 trials per condition and several sessions are pooled. We also require analytic fixed points that are stable across random seeds, which LFADS does not offer.

Gaussian CLDS is the method we directly extend, run both per session and pooled with shared readouts. The file gives no bits-per-spike number for it. We will rerun it on identical splits.

Three ablations each isolate one ingredient:

  • Pooled CLDS with a Gaussian likelihood but local readouts isolates the noise model.
  • Our likelihood fit separately per session isolates pooling.
  • Our model with a single noise family shared across all sessions isolates whether the noise model needs to be local.

Why not just…?

Why not just square-root transform the counts and run Gaussian CLDS? This is the most serious threat. At moderate-to-high firing rates the square-root transform roughly stabilizes the variance, and Gaussian CLDS with local readouts may then match us. We will report results binned by each neuron's mean rate. The proposal is only justified if the advantage concentrates in neurons below about 5 Hz, in calcium sessions, and in fixed-point bias on the synthetic data. If the transform matches us everywhere, the right conclusion is that only the local readout, not the likelihood, matters, and we will say so.

Why not just use LFADS with neural stitching? It already has every ingredient except closed-form fixed points and good small-data behavior. CLDS reports beating LFADS with 1–5 trials per condition in premotor cortex. Our trial-limited pooled experiment tests whether that advantage survives once both models can pool across sessions. If stitched LFADS closes the gap with pooling, our remaining claim reduces to interpretability: analytic fixed points against fixed points reverse-engineered after training.

Why not just fit one Poisson CLDS per session? That avoids every alignment problem. However, the cost of few trials is exactly what pooling is meant to address, and stitching's 0.18 $R^2$ gain in LFADS suggests pooling helps. The per-session ablation measures this directly.

Why not share the noise model too? It is simpler. But sessions differ in recording method and firing-rate regime. The shared-family ablation on mixed electrophysiology and calcium data tests whether letting each session keep its own noise family is necessary or merely harmless.

Proposed Experiments

The datasets are the following:

  • NLB '21 MC_Maze in its Large, Medium, and Small splits, for co-smoothing, behavioral decoding $R^2$, PSTH match, and forward prediction.
  • NLB DMFC_RSG, where the covariate is the timing interval. No reported result was found on Lacuna for any model's co-smoothing on DMFC_RSG.
  • The premotor center-out data and the head-direction thalamus data used by CLDS, for trial-limited co-smoothing and ring recovery. No numeric scores are reported on Lacuna.
  • A multi-session subset of the POYO primate corpus, to pool across real sessions with disjoint neurons.
  • The mouse striatal calcium data from CANDY, for the mixed-likelihood setting. The best reported figure there is decoding $R^2 \approx 0.7$ on an unseen mouse with 10% calibration data.

Three checks decide the outcome. The first is synthetic. With known fixed points and sessions of heterogeneous rates, fixed-point error under pooled Gaussian CLDS should grow as the share of low-rate or zero-inflated sessions grows, and it should stay flat under our model. If the error does not grow, the central claim is false.

The second check is on real data. Pooled co-smoothing in the 1–5 trials-per-condition regime should exceed both per-session Poisson CLDS and the square-root-transform Gaussian baseline, with the gap concentrated in low-rate neurons.

The third check is in the head-direction data. Fixed points estimated from pooled sessions should trace a single ring consistent across held-out sessions, and should be tighter than per-session estimates. A secondary check confirms that running the shared update from per-session sufficient statistics alone reproduces the centralized fit exactly, which establishes that the model can be run federated.