In classical systems neuroscience, the primary visual cortex (V1) was often treated as a sophisticated camera—a feedforward processor that transforms photons into edge detections and textures. However, if you show a mouse the exact same video twice, its neurons will not fire the same way twice. For decades, this variability was dismissed as "neural noise."
Recent research has overturned this view, revealing that a massive portion of what we called noise is actually a clear signal of the animal’s internal state. Whether an animal is running, whisking, or simply alert, these internal behavioral variables act as a powerful "gain control" that reshapes how sensory information is processed.
The Problem of the Moving Target
The central challenge in modern sensory modeling is that neural activity is a mixture of two distinct streams: external sensory input and internal behavioral modulation. As Daniel A. Butts notes in Data-Driven Approaches to Understanding Visual Neuron Activity, traditional models that ignore non-visual factors like attention and arousal leave a significant portion of neural variance unexplained.
Imagine trying to understand a movie while someone is constantly adjusting the brightness and contrast of the screen. If you don't know how the knobs are being turned, you cannot accurately describe the film. In the brain, these "knobs" are internal states like locomotion and arousal. Fabian H. Sinz and Paul G. Fahey demonstrated in Stimulus domain transfer in recurrent models that by simply tracking a mouse’s running speed and pupil dilation (a proxy for arousal), researchers could significantly improve their ability to predict how V1 neurons would respond to video.
Integrating Behavior into Sensory Models
To solve this, researchers have moved toward "multimodal" models. Instead of feeding a neural network only the visual pixels, they provide auxiliary channels for behavioral data. This approach was the cornerstone of the Sensorium competitions, described by Polina Turishcheva and colleagues in their Retrospective for the Dynamic Sensorium Competition.
By treating locomotion and pupil size as primary inputs alongside video, these models can account for the "gain" of the system. For instance, when a mouse starts running, the responsiveness of certain V1 neurons increases. A model that knows the mouse is running can "expect" this surge in activity, whereas a stimulus-only model would see it as inexplicable noise. Aiwen Xu and Cristopher Niell have shown that many neurons in V1 are actually "multitaskers," simultaneously encoding visual features and the animal's own movement (Multimodal Deep Learning Model Unveils Behavioral Dynamics).
Discovering Hidden Latent States
We cannot always measure every relevant internal variable. How do you measure "boredom," "focus," or "hunger" in a way that a computer can understand? Researchers are now using unsupervised learning to discover these "latent" states directly from the neural activity itself.
In Modeling Dynamic Neural Activity by combining Naturalistic Video Stimuli and Stimulus-independent Latent Factors, Finn Schmidt and colleagues developed a model that learns to separate stimulus-driven activity from internal fluctuations without being told what the behaviors are. Remarkably, the "hidden" factors the model discovered on its own correlated perfectly with biological realities like the animal's running speed.
Similarly, the TE-ViDS (Time-Evolving Visual Dynamical System) framework developed by Liwei Huang and Yonghong Tian uses a split-representation architecture to disentangle the temporal "flow" of the environment from the internal shifts of the brain (Time-Evolving Dynamical System for Learning Latent Representations). By isolating these internal states, we can finally see the "true" sensory representation underneath.
Why It Works: The Biological Intuition
This research direction suggests that the brain is not a passive receiver but a distributed, layered system, an idea championed as early as 1971 by Michael A. Arbib in Transformations and Somatotopy in Perceiving Systems.
The reason internal states dominate sensory cortex activity is functional: an animal needs to interpret the world differently depending on what it is doing. A flash of movement means something different to a cat that is hunting versus a cat that is grooming. By hardwiring behavioral signals directly into the sensory architecture—as Ana R. Inácio found in the specific long-range wiring of movement-tracking neurons (Distinct brain-wide presynaptic networks)—the brain ensures that its perception is always contextualized by its own actions and needs.
Understanding this modulation is the key to moving from "black box" neural models to true "digital twins" of biological vision. As Joel Bauer demonstrates in Movie reconstruction from mouse visual cortex activity, we can only truly "read the mind" of an organism when we account for the physical state of the body it inhabits.