Summary of Information Theoretic Quantities

Robin A. A. Ince, Stefano Panzeri, Simon R. SchultzEncyclopedia of Computational Neuroscience 2014 · DOI · DBLP

Abstract: This paper serves as a foundational primer for researchers applying information theory to neuroscience. Authors Robin A.A. Ince, Stefano Panzeri, and Simon R. Schultz distill complex mathematical concepts into a practical toolkit for quantifying how the nervous system processes and communicates information. By utilizing model-free metrics such as entropy and mutual information, researchers can analyze stimulus-response relationships without the restrictive assumptions of classical linear statistics.

Introduction

In the quest to decode the "language" of the brain, neuroscientists often encounter the limitations of traditional statistical methods. Correlation and variance, while useful, frequently assume linear relationships or specific data distributions, such as the Gaussian distribution. However, neural responses are inherently non-linear and noisy. In "Summary of Information Theoretic Quantities," Robin A.A. Ince, Stefano Panzeri, and Simon R. Schultz propose Information Theory as a superior framework for quantifying neural communication.

The Information-Theoretic Advantage

Information theory is fundamentally probabilistic and "model-free," meaning it is sensitive to any statistical relationship, not just linear ones. This framework allows researchers to ask precise, quantitative questions: How much does a single neural trial reduce our uncertainty about the stimulus? How much information is shared between a population of neurons?

By using the "bit" as a universal unit of measurement, information theory enables direct comparisons across disparate experiments—for instance, comparing the information processing capacity of a visual neuron to that of a tactile neuron. This universality is a significant advantage over variance-based metrics, which are often tied to the specific scale of the data.

Core Mathematical Building Blocks

The paper defines several fundamental quantities that serve as the building blocks for neural data analysis:

Shannon Entropy ($H$)

Shannon Entropy is a measure of the uncertainty associated with a discrete random variable. The authors describe it as a generalized form of variance. While variance is a second-order statistic meaningful primarily for unimodal distributions, entropy provides a meaningful measure of "spread" or "uncertainty" for any distribution shape, including multi-modal ones.

Mutual Information ($I$)

Mutual Information is a symmetric measure of the dependence between two variables, such as a stimulus ($S$) and a neural response ($R$). It is calculated as the difference between the total entropy and the conditional entropy ($I(R;S) = H(R) - H(R|S)$). A crucial insight provided by the authors is the symmetry of information: $I(R;S) = I(S;R)$. The information a response provides about a stimulus is mathematically identical to the information a stimulus provides about a response.

Kullback-Leibler (KL) Divergence

KL Divergence measures the "distance" (though not a formal metric) between two probability distributions. In neuroscience, this is often used to quantify how much a joint distribution of stimulus and response deviates from a model of independence. If two variables are independent, their Mutual Information—which is the KL divergence between the joint distribution and the product of the marginal distributions—is exactly zero.

Advanced Metrics and Neural Ensembles

The authors extend these basic concepts to handle the complexity of modern neuroscience data:

  • Conditional Mutual Information ($I(R; S_1 | S_2)$): This is essential when stimulus features are correlated. It allows researchers to determine if a neuron is truly coding for feature $S_1$ or if it only appears to do so because $S_1$ is correlated with $S_2$.
  • Multi-information: Used for analyzing neural ensembles, this metric quantifies the total amount of dependency within a group of neurons, helping to identify population-level coding strategies.
  • Information Rates: For continuous, time-varying dynamic processes, information rates provide a way to quantify the flow of information over time, which is critical for understanding sensory processing in real-time.

Conceptual Insights

One of the paper's strengths is its use of intuitive examples to ground abstract math. For instance, using a six-sided die, the authors demonstrate that being told a result is "even" reduces the possible outcomes from six to three. This reduction in uncertainty by a factor of two equates exactly to 1 bit of information.

Furthermore, the authors clarify the relationship between discrete and continuous variables. While Mutual Information and KL Divergence generalize smoothly to continuous spaces, "differential entropy" (the continuous version of entropy) possesses unique properties—such as potentially being negative—that can make it less intuitive in certain neural contexts compared to its discrete counterpart.

Limitations and Practical Considerations

Despite the elegance of these mathematical formulas, the authors acknowledge a significant hurdle: the estimation problem. Calculating these quantities from limited, noisy, and high-dimensional neural data is a major challenge. Small sample sizes often lead to biased estimates of information, where the data appears to contain more information than it actually does. The authors point to specialized bias-correction techniques and estimation strategies as necessary tools for practical implementation, which are discussed in depth in their companion literature.

Conclusion

"Summary of Information Theoretic Quantities" provides a concise yet comprehensive roadmap for applying information theory to neuroscience. By shifting the focus from linear correlations to the reduction of uncertainty, this framework offers a powerful, distribution-independent toolkit for uncovering the principles of neural coding and communication.

Summary updated

BibTeX
@inproceedings{ince2014summary,
  author = {Robin A. A. Ince and Stefano Panzeri and Simon R. Schultz and Simon R. Schultz},
  title = {{Summary of Information Theoretic Quantities}},
  booktitle = {Encyclopedia of Computational Neuroscience},
  year = {2014},
  doi = {10.1007/978-1-4614-7320-6\_306-1},
}

Loading related content...

Figure