Summary of Information Theoretic Quantities

Robin A. A. Ince, Stefano Panzeri, Simon R. SchultzEncyclopedia of Computational Neuroscience 2014 · OpenReview

TL;DR

This paper provides a clear and accessible overview of fundamental information-theoretic quantities like entropy and mutual information, explaining their relevance and application in neuroscience. It serves as a foundational guide for researchers looking to quantify information processing in complex systems, particularly neural activity.

What Problem Does This Paper Address?

The paper addresses the challenge of precisely quantifying information flow and processing within complex systems, with a particular focus on neuroscience. While the concept of "information" is intuitive, its technical definition and measurement require a rigorous framework. This paper aims to demystify and define key information-theoretic quantities, making them accessible to researchers who may not have a deep background in information theory but need to apply these concepts to their work, such as understanding how neural responses encode information about stimuli.

Methods

The paper introduces and defines several core information-theoretic quantities. It starts with Shannon entropy, presented as a measure of uncertainty in a random variable, derived from its probabilistic basis. It then moves to mutual information, which quantifies the reduction in uncertainty about one variable given knowledge of another. The paper presents mutual information in three equivalent forms, highlighting its interpretation as a measure of dependence and its symmetry. It also introduces the Kullback-Leibler (KL) divergence as a measure of difference between probability distributions and shows how mutual information can be expressed as a KL divergence between the joint distribution and the independent model. Finally, it briefly touches upon multi-information and conditional mutual information for multivariate scenarios, and entropy and information rates for continuous, dynamic processes.

Datasets

Not discussed in the paper.

Key Insights and Ideas

  • Entropy as a Measure of Uncertainty: Entropy provides a general, non-parametric way to quantify the variability or uncertainty within any probability distribution, going beyond traditional statistical measures like variance.
  • Mutual Information as Shared Information: Mutual information elegantly captures the "shared information" between two variables, quantifying how much knowing one variable reduces uncertainty about the other. This is crucial for understanding how neural responses encode stimulus information.
  • Information Theory's Broad Applicability: The framework is presented as universally applicable, sensitive to any relationship (linear or non-linear) and providing meaningful units for cross-experiment comparisons.
  • Single-Trial Analysis: A key advantage highlighted is the ability to use these quantities to study information gained from single experimental trials, rather than relying solely on averaged responses, which can obscure important dynamic information.
  • Information as Deviation from Independence: Mutual information can be understood as a measure of how far two variables deviate from statistical independence, making it a powerful tool for detecting relationships.

Key Results

The paper's primary "result" is the clear definition and explanation of key information-theoretic quantities. It demonstrates through examples (like the die roll) how these measures work and what they represent. For instance, it shows that knowing a die roll is even reduces the uncertainty by 1 bit, illustrating the concept of information gain. It also mathematically establishes the equivalence of different formulations of mutual information and its relationship to KL divergence.

Limitations and Future Work

The paper acknowledges that while differential entropy is the continuous analogue of discrete entropy, it has not been widely used in neuroscience and is therefore beyond the scope of the article. It also notes that while mutual information generalizes to multiple stimuli, the concept of discriminability's generalization is less clear. The paper points to other entries for more details on estimation techniques ("Estimation of Information-Theoretic Quantities") and applications ("Applications of Information-Theoretic Quantities in Neuroscience"), suggesting these are areas for further exploration.

Summary updated

BibTeX
@inproceedings{ince2014summary,
  author = {Robin A. A. Ince and Stefano Panzeri and Simon R. Schultz},
  title = {{Summary of Information Theoretic Quantities}},
  booktitle = {Encyclopedia of Computational Neuroscience},
  year = {2014},
  url = {https://openreview.net/forum?id=Q29EnUmmSE},
}
Figure