Biological systems face a fundamental challenge: transmitting and storing information reliably through noisy, resource-constrained physical substrates. Whether considering a network of spiking neurons in the visual cortex or a cluster of interacting genes in a developing embryo, the underlying mechanisms often appear complex and heuristic.
However, a powerful organizing framework treats these biological mechanisms not as arbitrary evolutionary accidents, but as mathematical consequences of a single global objective: maximizing the preservation of Shannon information. By framing synapses and genetic pathways as communication channels, we can derive observed biological phenomena—such as Spike-Timing Dependent Plasticity (STDP)—directly from first principles.
The Core Intuition: The Communication Channel View
To understand why a neuron or a genetic network adapts the way it does, we can model it as a classic Shannon communication channel. The system receives an input signal $X$ (such as presynaptic spike trains or transcription factor concentrations) and maps it to an output response $Y$ (such as postsynaptic spikes or gene expression levels).
The goal of the system is to maximize the Mutual Information $I(X; Y)$ between the input and the output. Mathematically, mutual information is defined as:
$$I(X; Y) = H(Y) - H(Y|X)$$
Where:
- $H(Y)$ is the entropy of the output. It represents the diversity or capacity of the output states.
- $H(Y|X)$ is the conditional entropy of the output given the input. It represents the noise, uncertainty, or variability in the system.
This decomposition reveals two distinct strategies a biological system can use to optimize its information transmission capacity:
- Maximizing Output Entropy $H(Y)$: Ensuring the system uses its full dynamic range and does not saturate or waste resources on redundant representations.
- Minizing Conditional Entropy $H(Y|X)$: Reducing internal noise, temporal jitter, and variability to make the output highly reliable and predictable given a specific input.
The foundational "Infomax" principle, introduced by Ralph Linsker in Towards an Organizing Principle for a Layered Perceptual Network, posits that networks optimize the preservation of Shannon information as signals transition between successive layers. This framework allows us to understand synaptic plasticity as a gradient ascent process on the network's information transmission capacity.
Input X ───► [ Synaptic Weights / Channel Parameters ] ───► Output Y
▲
│ (Gradient Ascent on I(X;Y))
└─ Infomax Optimization
Maximizing Output Entropy: Dynamic Range and Adaptation
To maximize $H(Y)$, a neuron must adjust its parameters so that its output distribution matches the statistical structure of its environment. If a neuron fires at its maximum rate all the time, or if it rarely fires at all, its output entropy is low, and it transmits very little information.
Adorján et al. demonstrated this strategy in the context of contrast adaptation in the visual cortex. By deriving a non-Hebbian learning rule via stochastic gradient ascent on the Infomax objective, the authors showed that adjusting the probability of neurotransmitter release at synapses maximizes the entropy of the output distribution relative to the input. This slow adaptation mechanism ensures that the neural response remains highly sensitive to input variations across a wide dynamic range, preventing saturation when lighting conditions change.
At the structural level, Brunel explored how optimizing information transmission per synapse shapes the distribution of weights. By tuning output firing distributions using Shannon mutual information, this optimization identifies the ideal balance between the number of stored patterns and the information content of each pattern, explaining why biological structures like the cerebellum contain a high proportion of "silent" or weakly active synapses.
Minimizing Conditional Entropy: Reducing Noise and Jitter
The second half of the information-theoretic objective requires minimizing $H(Y|X)$, which translates to maximizing the reliability of the system. If the same input $X$ produces wildly different outputs $Y$ due to internal noise, information is lost.
Bohte et al. proposed that synaptic plasticity rules can emerge entirely from the objective of minimizing neural noise. By implementing a gradient descent process on the conditional entropy of a neuron's output, the authors showed that connections are naturally adjusted to make a neuron's output more reliable and less variable. As further detailed by Bohte et al., minimizing conditional entropy forces the neuron to suppress redundant or uncertain firing, ensuring that spikes occur with high certainty in response to specific input patterns.
In spiking networks, minimizing conditional entropy directly corresponds to reducing temporal jitter. When synapses are optimized for maximum information transfer, they narrow the window of uncertainty in spike timing. As demonstrated in Optimal context separation of spiking haptic signals by second-order somatosensory neurons, this reduction in temporal jitter allows a small population of neurons to represent high-dimensional, rapid tactile stimuli with sub-millisecond precision, relying on the precise timing of the very first spikes rather than slow averages.
Deriving Spike-Timing Dependent Plasticity (STDP)
Perhaps the most compelling evidence for this framework is that it derives the asymmetric temporal window of Spike-Timing Dependent Plasticity (STDP)—the biological rule where a synapse is strengthened if a presynaptic spike precedes a postsynaptic spike, and weakened if it follows it—directly from information theory.
Synaptic Change (Δw)
▲
│ *
│ * * (Potentiation: Pre before Post)
│ * *
──────────┼────────────────────────► Δt (t_post - t_pre)
│ * *
│ * * (Depression: Post before Pre)
│ *
Deriving online, spike-based rules from global information objectives is mathematically challenging because the relationship between precise spike timings and mutual information is highly non-linear. Chechik et al. overcame this by applying the Infomax principle to a spiking neuron model, proving that treating synaptic weight updates as a gradient ascent process aimed at maximizing mutual information directly yields the classic asymmetric STDP learning curve.
Toyoizumi et al. expanded on this, demonstrating that the specific asymmetric temporal window of STDP represents an mathematically optimal strategy for maximizing mutual information between input and output spike trains under physical constraints. Rather than being an arbitrary biological quirk, the shape of the STDP window is precisely tuned to optimize information transmission efficiency.
To bridge the gap between abstract objectives and local biological implementation, Legenstein et al. formulated a Lagrangian objective function that balances the maximization of mutual information against a homeostasis constraint using Kullback-Leibler divergence. Performing gradient ascent on this global objective reveals that classical biological mechanisms—like sliding-threshold rules—are mathematically optimal for maintaining stable target firing rates while extracting independent components.
Furthermore, this information-maximizing framework accounts for the diversity of plasticity rules found in nature:
- Complex Plasticity Windows: Hennequin et al. demonstrated that "triplet-based" STDP rules act as a biological approximation of an ideal information-maximizing algorithm in neurons with spike-frequency adaptation, explaining why different neuron types exhibit distinct learning windows.
- Hardware Matching: Miyata et al. and Miyata et al. showed that achieving high information fidelity in associative memory requires a precise match between how neurons shift their timing (their Phase Response Curve) and how synapses change their strength (STDP). The observed spatial variation of plasticity rules across different dendritic regions is a functional adaptation to maximize mutual information during memory retrieval.
- State-Dependent Reorganization: Yoshida et al. showed that maximizing mutual information while penalizing metabolic expenditure derives the exact state-dependent synaptic changes observed during non-rapid eye movement (NREM) sleep, providing an information-theoretic account of memory consolidation and pruning.
Universality Across Biological Scales: From Neurons to Genes
The principle of channel capacity optimization is not unique to neural circuitry; it operates as a universal organizing principle across diverse biological scales, including molecular signaling and genetic regulatory networks (GRNs).
Tkacik et al. and Walczak et al. modeled genetic regulatory networks as communication channels where the mutual information between an input chemical signal concentration and the output expression levels of multiple genes is maximized. By treating the network's topology and binding affinities as parameters to be optimized, the authors demonstrated that cells can reach the physical limits of control possible in a noisy, stochastic environment.
As reviewed by Walczak in Information transmission in genetic regulatory networks: a review, applying variational calculus to maximize mutual information allows researchers to identify the optimal distribution of inputs and network parameters. This approach reveals that biological circuits—such as those responsible for spatial patterning in developing fly embryos—often operate near their theoretical "channel capacity."
To compute these theoretical upper bounds, researchers treat these systems as constrained optimization problems. For instance, Einolghozati et al. calculated the capacity of diffusion-based molecular communication by applying iterative numerical algorithms to find the optimal probability distribution of input signals. Similarly, Cepeda-Humerez et al. employed the Blahut-Arimoto algorithm to quantify the maximum information transmission capacity of signaling channels, demonstrating how complex, multi-step regulatory procedures like stochastic proofreading act to suppress error and alleviate crosstalk.
Connecting Biology to Machine Learning
Beyond explaining biological phenomena, information-theoretic plasticity provides a theoretical bridge to unsupervised machine learning. Modern artificial neural networks rely on global optimization algorithms like backpropagation, which require non-local information unavailable to biological synapses. Biological learning must rely on local rules, such as Hebbian plasticity.
Liu et al. established a formal mathematical connection between these two domains. The authors proved that maximizing a learning objective based directly on biological Hebbian synaptic plasticity rules is mathematically equivalent to increasing the lower bound on the mutual information between input data and learned representations.
This equivalence provides a profound insight into why the brain works the way it does: local, biologically plausible synaptic updates are not primitive approximations of global optimization algorithms. Instead, they are mathematically optimized local engines designed to maximize information efficiency, allowing distributed networks to learn high-quality, abstract features without the need for global supervision.
Key Foundational Work
Analytic Solutions to the Formation of Feature-Analysing Cells of a Three-Layer Feedforward Visual Information Processing Neural Net (1989) — contributed 1 concept elements to this direction.
Towards an Organizing Principle for a Layered Perceptual Network (1987) — contributed 1 concept elements to this direction.