Abstract
Sensory neurons often have variable responses to repeated presentations of the same stimulus, which can significantly degrade the stimulus information contained in those responses. This information can in principle be preserved if variability is shared across many neurons, but depends on the structure of the shared variability and its relationship to sensory encoding at the population level. The structure of this shared variability in neural activity can be characterized by latent variable models, although they have thus far typically been used under restrictive mathematical assumptions. Here we introduce two nonlinear latent variable models for analyzing large-scale neural recordings. We first present a general nonlinear latent variable model that is agnostic to the stimulus tuning properties of the individual neurons, and is hence well suited for exploring neural populations whose tuning properties are not well characterized. This motivates a second class of model, the Generalized Affine Model, which simultaneously determines each neuron's stimulus selectivity and a set of latent variables that modulate these stimulus-driven responses both additively and multiplicatively. While these approaches can detect very general nonlinear relationships in shared neural variability, we find that neural activity recorded in anesthetized primary visual cortex (V1) is best described by a single additive and single multiplicative latent variable, i.e. an 'affine model'. In contrast, application of the same models to recordings in awake macaque prefrontal cortex discover more general nonlinearities to compactly describe the population response variability. These results thus demonstrate how nonlinear latent variable models can be used to describe population response variability, and suggest that a range of methods is necessary to study different brain regions under different experimental conditions.
Introduction
Neurons in the brain are notoriously "noisy," responding differently to the exact same stimulus across multiple trials. Crucially, this variability is often shared across many neurons (known as noise correlations), which can either hinder or help the brain's ability to encode information. To study this, researchers use latent variable models to identify hidden "factors" that explain the activity of hundreds of neurons.
However, traditional methods like Principal Component Analysis (PCA) or Factor Analysis (FA) assume these relationships are linear. Given the inherent nonlinearities of neural circuits—such as thresholding and saturation—linear models often fail to capture the true structure of population activity. This paper introduces nonlinear models that are flexible enough to fit complex data while remaining structured enough to be interpretable.
Methods
The authors propose two frameworks based on autoencoder-style neural networks:
- Stacked Rectified Latent Variable Model (SRLVM): A "stimulus-agnostic" model. It uses deep neural networks to encode high-dimensional neural activity into a low-dimensional latent space and then decode it back into predicted firing rates. Because it does not explicitly incorporate stimulus information, it must capture both stimulus-driven tuning and internal variability within the latents.
- Generalized Affine Model (GAM): A more structured approach that explicitly incorporates stimulus tuning. It models a neuron's response as a combination of its stimulus-driven response modulated by shared latent variables. Specifically, it allows for multiple multiplicative (gain-like) and additive (offset-like) latent factors.
Both models are fit by maximizing log-likelihood under a Gaussian noise assumption (using square-rooted spike counts) and utilize $L_2$ regularization to prevent overfitting. The models were validated on recordings from the primary visual cortex (V1) of anesthetized macaques and the prefrontal cortex (PFC) of awake macaques.
Key Results and Insights
The "Affine" Nature of V1
One of the most striking findings is that in anesthetized V1, a very simple model—one additive and one multiplicative latent variable—captures almost as much variance as the most complex deep neural network. This suggests that V1 variability under anesthesia is dominated by simple, global fluctuations in gain and offset. The Quality Index (QI) difference between the simple affine model and the complex SRLVM was negligible in this region.
PFC Complexity
In contrast, the awake PFC is much more "high-dimensional." The simple affine model fails significantly here. To match the performance of the general SRLVM, the GAM requires many more latent variables (up to 9 additive and 9 multiplicative factors). The SRLVM's ability to outperform simpler models in certain regimes suggests that there are nonlinear interactions between latents and stimuli that go beyond simple scaling or shifting.
Discussion
These results demonstrate how nonlinear latent variable models can be used to describe population response variability. While V1 under anesthesia appears relatively simple, the awake PFC exhibits complex nonlinear interactions. It remains unclear if this difference is due to the brain region itself or the cognitive demands of the awake state. Future work could extend these models to account for within-trial temporal dynamics or utilize fully probabilistic frameworks like Variational Autoencoders for better uncertainty quantification.
Summary updated