Disentangling Internal States from Sensory Cortical Responses

20 concepts15 papers1 proposal

If you present an animal with the exact same visual stimulus multiple times, the neurons in its primary visual cortex will fire differently on every single trial. For decades, systems neuroscience treated this trial-to-trial variability as stochastic "noise"—an inherent imperfection of biological hardware that could only be overcome by averaging neural responses over many repetitions.

However, recent research reveals that this variability is not random noise at all. Instead, it represents a highly structured, ongoing conversation within the brain. Sensory cortices do not operate in a vacuum; they are continuously modulated by the animal's internal states, behavioral actions, and fluctuating arousal levels. By learning to model and disentangle these internal dynamics from external sensory inputs, researchers are unlocking unprecedented accuracy in predicting brain activity and dramatically improving the performance of brain-machine interfaces.

The Illusion of Cortical Noise

To understand why sensory responses vary, we have to look at the background activity of the neural network itself. The brain is constantly generating endogenous fluctuations independent of any external input.

As Cui et al. (2016) demonstrated, much of the apparent "noise" in the macaque visual cortex is actually predictable network activity captured by local field potentials (LFPs). By integrating these ongoing LFP signals into a predictive model, the authors achieved a fivefold improvement in predicting spike responses compared to models relying on the visual stimulus alone. This indicates that what looks like trial-to-trial variability is often the deterministic signature of an internal cortical state.

[Visual Stimulus] -------> [ Sensory Cortex ] <------- [ Internal Brain State ]
                                   |                     (Arousal, Locomotion, LFPs)
                                   v
                        [ Variable Neural Response ]

These internal states are driven by deep neuromodulatory systems. For example, a dynamical systems model developed by Safaai et al. (2014) showed that ongoing activity in the locus coeruleus—a brainstem nucleus responsible for physiological arousal—directly drives fluctuations in the sensory cortex. Accounting for this subcortical drive nearly doubled the amount of extractable information about sensory stimuli, proving that background physiological fluctuations act as a powerful modulator of sensory processing.

The Multi-Tasking Sensory Cortex

In awake, behaving animals, these internal states manifest as overt behaviors. When an animal runs, moves its eyes, or experiences a surge in alertness, its entire cortex is flooded with non-sensory signals.

This means that neurons in the primary visual cortex (V1) are not simple feature detectors; they are multiplexers. Xu et al. (2023) showed that in freely moving mice, the vast majority of V1 neurons simultaneously process both visual inputs and behavioral contexts, such as head orientation and running speed.

When computational models ignore these non-visual factors, their predictive power hits a strict ceiling. Butts (2019) notes that traditional, purely stimulus-driven frameworks leave a massive portion of neural variance unexplained because they omit variables like eye movements, attention, and metabolic states. Work by Sinz et al. (2018) and Turishcheva et al. (2024) confirmed that incorporating behavioral tracking data—such as pupil dilation and locomotion—directly into deep learning models is essential for accurately predicting how the cortex responds to complex, naturalistic videos.

The Danger of Omitting the Internal State

Failing to account for internal state modulation does more than just lower a model's accuracy; it can lead to fundamental misinterpretations of how the brain changes over time.

Consider the phenomenon of "representational drift," where the neural population response to a fixed stimulus gradually changes over days or weeks. Sadeh (2022) demonstrated that a significant portion of this observed drift is actually caused by unmodeled behavioral variability. If an animal's average arousal level or movement patterns shift across experimental sessions, a stimulus-only analysis will mistake this behavioral shift for a rewiring of the sensory representation itself.

Similarly, VanRullen et al. (2018) highlighted that endogenous factors like alertness and vigilance frequently confound efforts to isolate the role of neural oscillations (such as alpha rhythms) in perception. Without explicitly decoupling internal states from the analysis, it is impossible to determine whether a neural signal is a direct driver of behavior or merely a byproduct of a global state change.

How Modern Models Disentangle the Signals

To isolate the true relationship between external stimuli and neural activity, modern computational neuroscience employs two main strategies: explicit behavioral tracking and latent variable inference.

Explicit Behavioral Tracking

The most straightforward approach is to measure behavioral variables directly and feed them into the model as auxiliary inputs. Models like those evaluated by Conwell et al. (2021) integrate multi-modal signals—such as whisking, running speed, and pupil size—alongside visual features to predict cortical responses with high biological fidelity.

Latent Variable Inference

However, we cannot manually record every internal factor; states like attention, hunger, or micro-arousals are often hidden. To capture these, researchers use unsupervised machine learning to infer "latent factors" directly from the simultaneous firing of large populations of neurons.

A probabilistic model presented in Modeling Dynamic Neural Activity by combining Naturalistic Video Stimuli and Stimulus-independent Latent Factors demonstrates the power of this approach. By learning hidden, stimulus-independent factors from population activity, the model not only achieved state-of-the-art predictive accuracy but also automatically discovered the animal's running speed and cortical anatomy without ever being explicitly given that data.

Taking this a step further, the Time-Evolving Visual Dynamical System (TE-ViDS) developed by Huang et al. (2025) uses a split-representation architecture. It explicitly separates stimulus-driven signals from spontaneous internal brain states, treating the internal states as "nuisance variables" that can be factored out to dramatically improve the decoding of natural scenes from neural spikes.

[ Population Spikes ] ---> [ Unsupervised Latent Model ] ---> Disentangled States:
                                                                ├── Stimulus Representation
                                                                └── Latent Internal State (Arousal)

Applications: From BMIs to Foundation Models

Disentangling internal states is not just an exercise in better system identification; it has profound practical implications for neurotechnology.

In Brain-Machine Interfaces (BMIs), trial-to-trial variability is a major hurdle. If a user's baseline neural excitability fluctuates, a static BMI decoder will misinterpret those changes as intentional commands, causing the system to become unstable. Safaai et al. (2016) proposed that accounting for this fluctuating excitability could drastically improve BMI reliability. Putting this into practice, De Feo et al. (2017) developed a state-dependent decoding algorithm that "discounts" background pre-stimulus activity, achieving a 22% increase in extracted information and creating more stable device control without requiring additional electrodes.

Finally, this paradigm shift is enabling the creation of large-scale neural foundation models. By training on massive datasets across multiple animals, models like the one introduced by Wang et al. (2025) learn a generalized representation of vision. Because these models explicitly account for and isolate behavioral modulations like arousal and locomotion, they can successfully generalize to entirely new animals and stimulus types, uncovering the fundamental, invariant principles of sensory transformation in the living brain.

Key Foundational Work

Implications of the Dependence of Neuronal Activity on Neural Network States for the Design of Brain-Machine Interfaces (2016) — contributed 2 concept elements to this direction.
A dynamical systems model of the effect of Locus Coeruleus firing on single trial cortical state dynamics (2014) — contributed 1 concept elements to this direction.
Inferring Cortical Variability from Local Field Potentials (2016) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_internal_state_modulation_5fa8bf2d,
  title = {internal state modulation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/internal-state-modulation/rdr_5fa8bf2dd9f64d94a5e30d432adbd000},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_internal_state_modulation_5fa8bf2d_v1,
  title = {internal state modulation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/internal-state-modulation/rdr_5fa8bf2dd9f64d94a5e30d432adbd000/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake