Model Independent Quantification of Neural Information

97 concepts87 papers1 proposal

Neuroscientists have long sought to understand the "language" of the brain. Historically, this was done by constructing tuning curves—plots that show how a neuron's average firing rate changes in response to an external variable, like the orientation of a visual stimulus or the position of an animal in space. While intuitive, this approach has a fundamental flaw: it assumes we already know what features the neuron cares about and how it encodes them. If a neuron encodes information through precise spike timing rather than raw firing rates, or if it responds to complex, non-linear combinations of features, traditional tuning curves will miss it entirely.

To solve this, researchers turn to information theory. By treating a neuron or a neural population as a communication channel, we can use Shannon's mutual information to measure exactly how many bits of information a neural response carries about the outside world. Crucially, this approach is model-independent and decoder-independent. It does not care whether the code is linear or non-linear, rate-based or temporal; it simply quantifies the reduction in uncertainty.

The Core Mechanism: Subtracting Noise from Total Entropy

To understand why information-theoretic quantification works, we must look at how information is calculated from real neural data. The most direct framework relies on a simple intuition: information is the difference between the total variability of a neuron's response and the variability that remains when the stimulus is held constant (the noise).

Mathematically, the mutual information $I(S; R)$ between a stimulus $S$ and a neural response $R$ is formulated as:

$$I(S; R) = H(R) - H(R|S)$$

Where $H(R)$ is the total entropy of the neural response, and $H(R|S)$ is the noise entropy.

To calculate these values empirically, researchers use what is known as the "direct method," pioneered in neural contexts by Charles F. Stevens et al. in Information through a Spiking Neuron. The continuous spike train of a neuron is discretized into small temporal bins (e.g., 1 or 2 milliseconds), where a '1' represents a spike and a '0' represents silence. By grouping adjacent bins together, the spike train is transformed into a sequence of binary "words."

Spike Train:  ---|---|-------|---|---
Discretized:  [1] [1] [0] [0] [1] [1]
Binary Word:  110011
  1. Total Entropy $H(R)$: The authors present a wide variety of diverse, dynamic stimuli to the organism and record the resulting words. $H(R)$ measures the overall diversity or vocabulary size of the neuron's response. If a neuron always produces the same few words, its total entropy is low.
  2. Noise Entropy $H(R|S)$: The authors repeat the exact same stimulus sequence dozens of times. Because biology is stochastic, the neuron will not respond identically every time. $H(R|S)$ quantifies this trial-to-trial variability. If the neuron responds identically across all trials, the noise entropy drops to zero.

By subtracting the noise entropy from the total entropy, as also detailed by Kamal Sen in Spike timing and the coding of naturalistic sounds in a central auditory area of songbirds, we isolate the reproducible, structured variations that are genuinely driven by the stimulus. This allows researchers like Huan Wang et al. in Speech coding and information processing by auditory neurons to determine exactly how many bits of data a single spike or temporal pattern transmits.

Decoder Independence and Behavioral Relevance

A common trap in computational neuroscience is using an external machine learning model (a decoder) to predict a stimulus from neural activity, and using the decoder's accuracy as a proxy for information content. The danger here is that an advanced artificial decoder might extract information that the brain itself cannot actually read or utilize.

To bypass this bias, Giuseppe Pica et al. introduced a decoder-independent metric called Intersection Information ($I_{II}$) in Quantifying how much sensory information in a neural code is relevant for behavior. Instead of looking at the neuron and the stimulus in isolation, this approach evaluates a three-way intersection between the Stimulus ($S$), the Neural Response ($R$), and the organism's behavioral Choice ($C$).

   Stimulus (S)       Neural Response (R)
         \                 /
          \               /
           [Intersection] <-- True Behavioral Code
                  |
              Choice (C)

Intersection Information isolates the specific subset of bits that are simultaneously informative about the stimulus and statistically linked to the final decision. If a neuron changes its firing pattern based on a stimulus change, but that variation has zero statistical relationship with whether the animal turns left or right, $I_{II}$ reveals that this information is discarded by downstream brain regions. This framework allows for an objective assessment of which neural features (such as millisecond-scale spike timing versus coarse firing rates) constitute the true "behavioral code."

Dissecting Population Codes: Synergy and Redundancy

When moving from a single neuron to a population of thousands, information quantification faces a scaling challenge: neurons are rarely independent. They are often driven by shared inputs or correlated internal noise.

To evaluate how efficiently a population represents data, Sheila Nirenberg demonstrated a rigorous method in Synergy, Redundancy, and Independence in Population Codes, Revisited. By calculating the Kullback-Leibler (KL) divergence between the true joint probability distribution of the population's responses and a factored distribution that assumes complete independence, one can precisely calculate $\Delta I$—the exact amount of information lost when downstream decoders ignore noise correlations.

Furthermore, standard pairwise mutual information fails when network interactions become highly multivariate, involving feedback loops and divisive normalization. To capture these complex dynamics, Greg Ver Steeg et al. demonstrated the utility of Total Correlation (TC) in Functional Connectivity via Total Correlation: Analytical results in Visual Areas. Total Correlation generalizes mutual information to arbitrarily large groups of variables, mapping how information is distributed across entire networks without suffering from the mathematical confounds of simpler pairwise metrics.

From Biology to Artificial Networks

The power of model-independent information quantification extends beyond biological brains into artificial neural networks (ANNs) and spiking neural networks (SNNs). For instance, Marius Köppel showed in Invariant Representations with Stochastically Quantized Neural Networks that by stochastically quantizing network activations into discrete random variables, researchers can compute exact mutual information using closed-form analytical formulas. This eliminates the need for unstable variational approximations or adversarial training when tracking information flow through deep learning layers.

Whether applied to a rat's barrel cortex processing whisker vibrations, as explored by Ehsan Arabzadeh et al. in Deciphering the Spike Train of a Sensory Neuron: Counts and Temporal Patterns in the Rat Whisker Pathway, or to an energy-efficient neuromorphic architecture, information-theoretic quantification works because it respects a fundamental truth: information is fundamentally about the reduction of uncertainty, measured in bits, independent of the physical medium or mathematical model carrying it.

Key Foundational Work

An Information-Theoretic Approach to Deciphering the Hippocampal Code (1992) — contributed 3 concept elements to this direction.
Information through a Spiking Neuron (1995) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_mutual_information_quantification_184beef2,
  title = {mutual information quantification},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/mutual-information-quantification/rdr_184beef26c6646169e4e0418d32c535a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_mutual_information_quantification_184beef2_v1,
  title = {mutual information quantification},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/mutual-information-quantification/rdr_184beef26c6646169e4e0418d32c535a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake