Synergy, Redundancy, and Independence in Population Codes, Revisited

Peter E. Latham, Sheila NirenbergJournal of Neuroscience 2005 · DOI

Abstract

Decoding the activity of a population of neurons is a fundamental problem in neuroscience. A key aspect of this problem is determining whether correlations in the activity, i.e., noise correlations, are important. If they are important, then the decoding problem is high dimensional: decoding algorithms must take the correlational structure in the activity into account. If they are not important, or if they play a minor role, then the decoding problem can be reduced to lower dimension and thus made more tractable. The issue of whether correlations are important has been a subject of heated debate. The debate centers around the validity of the measures used to address it. Here, we evaluate three of the most commonly used ones: synergy, $\Delta I_{shuffled}$, and $\Delta I$. We show that synergy and $\Delta I_{shuffled}$ are confounded measures: they can be zero when correlations are clearly important for decoding and positive when they are not. In contrast, $\Delta I$ is not confounded. It is zero only when correlations are not important for decoding and positive only when they are; that is, it is zero only when one can decode exactly as well using a decoder that ignores correlations as one can using a decoder that does not, and it is positive only when one cannot decode as well. Finally, we show that $\Delta I$ has an information theoretic interpretation; it is an upper bound on the information lost when correlations are ignored.

Introduction

When a population of neurons responds to a stimulus, their firing patterns are often correlated. A central question in neuroscience is whether these "noise correlations"—the tendency of neurons to fire together beyond what is explained by the stimulus—are a critical part of the neural code. If correlations are important, decoding the brain's activity is a high-dimensional challenge; a downstream "reader" neuron would need to know the intricate joint distribution of the entire population. If they aren't important, the problem simplifies significantly: we can treat neurons as independent encoders. The field has been stuck in a "heated debate" because different mathematical measures often yield contradictory results. This paper evaluates which measures actually determine if correlations matter for decoding.

Evaluating Information-Theoretic Metrics

The authors use an information-theoretic framework to evaluate three metrics:

  1. Synergy/Redundancy: Compares the information in a population to the sum of information in individual neurons.
  2. Shuffled Information ($\Delta I_{shuffled}$): Compares the information in the true response to the information in a "shuffled" version where correlations are removed.
  3. $\Delta I$: A measure based on the Kullback-Leibler (KL) distance between an optimal decoder (which knows the correlations) and an "independent" decoder (which ignores them).
Comparison of joint and independent response distributions.

As illustrated in Figure 1, the independent model (b) is a simplified representation that ignores the correlational structure present in the true joint distribution (a). The metric $\Delta I$ specifically quantifies the impact of this simplification on decoding performance.

The Failure of Synergy and Shuffled Information

One of the paper's most significant findings is that synergy and $\Delta I_{shuffled}$ are "confounded" measures. They can be positive even when correlations are demonstrably irrelevant for decoding. For example, $\Delta I_{shuffled}$ measures information in a distribution (the shuffled one) that the brain never actually encounters. Consequently, it can be non-zero even when correlations don't matter at all.

Geometric scenarios illustrating how correlations can be irrelevant for decoding.

Figure 6 demonstrates the "Correlation $\neq$ Importance" principle. In panel (a), although the responses are 100% correlated (indicated by the lines), the stimuli $s_1$ through $s_4$ are perfectly separable in the response space. A decoder ignoring these correlations would still achieve perfect accuracy. In such cases, synergy and $\Delta I_{shuffled}$ can be misleadingly positive, whereas $\Delta I$ correctly remains zero.

$\Delta I$ as the Rigorous Standard

The authors prove that $\Delta I$ is a rigorous upper bound on the information lost when correlations are ignored. If $\Delta I$ is small (e.g., 10% of total information), then a decoder ignoring correlations will lose at most 10% of the available information. Practical applications in systems like the retina suggest that $\Delta I/I$ is often low, implying that for many biological systems, ignoring correlations might be a "good enough" strategy for the brain.

Conclusion and Future Directions

While $\Delta I$ is theoretically superior, it is difficult to estimate from real neural data due to the high dimensionality of joint distributions. The authors suggest that instead of direct estimation, researchers should build actual decoders—one that ignores correlations and one that doesn't—and compare their performance on real tasks. This "brute force" approach is often more tractable and provides a direct measure of the importance of population structure.

Summary updated

Abstract

Decoding the activity of a population of neurons is a fundamental problem in neuroscience. A key aspect of this problem is determining whether correlations in the activity, i.e., noise correlations, are important. If they are important, then the decoding problem is high dimensional: decoding algorithms must take the correlational structure in the activity into account. If they are not important, or if they play a minor role, then the decoding problem can be reduced to lower dimension and thus made more tractable. The issue of whether correlations are important has been a subject of heated debate. The debate centers around the validity of the measures used to address it. Here, we evaluate three of the most commonly used ones: synergy, DeltaI(shuffled), and DeltaI. We show that synergy and DeltaI(shuffled) are confounded measures: they can be zero when correlations are clearly important for decoding and positive when they are not. In contrast, DeltaI is not confounded. It is zero only when correlations are not important for decoding and positive only when they are; that is, it is zero only when one can decode exactly as well using a decoder that ignores correlations as one can using a decoder that does not, and it is positive only when one cannot decode as well. Finally, we show that DeltaI has an information theoretic interpretation; it is an upper bound on the information lost when correlations are ignored.
BibTeX
@article{latham2005synergy,
  author = {Peter E. Latham and Sheila Nirenberg},
  title = {{Synergy, Redundancy, and Independence in Population Codes, Revisited}},
  journal = {Journal of Neuroscience},
  year = {2005},
  doi = {10.1523/jneurosci.5319-04.2005},
}
Figure