The foundational axiom of neural learning is Hebbian plasticity: "neurons that fire together, wire together." When a presynaptic neuron repeatedly contributes to the firing of a postsynaptic partner, the connection between them strengthens.
While this mechanism beautifully explains how brains map correlations in the environment, it contains a fatal mathematical flaw. Hebbian learning is a positive feedback loop. If a synapse strengthens, the presynaptic neuron becomes more effective at driving the postsynaptic neuron. This increases their co-firing rate, which triggers further synaptic strengthening.
In recurrent neural networks, where neurons are tightly interconnected via feedback loops, this process quickly spirals out of control. As noted by Toyoizumi et al. and Zenke et al., without an explicit balancing mechanism, Hebbian updates inevitably cause runaway excitation. The network is driven to pathological saturation—where every neuron fires at its maximum possible rate—or to complete non-functional silence.
This instability is not unique to simple rate models; it equally plagues Spike-Timing-Dependent Plasticity (STDP). Abbott and Chen et al. emphasize that associative learning based on precise spike timing causes unbounded synaptic growth if left unregulated. Biologically realistic architectures, such as the supralinear networks studied by Lengyel et al., lack the artificial saturation functions (like sigmoids) common in deep learning, leaving them exceptionally vulnerable to explosive electrical activity during learning.
The Temporal Paradox of Slow Homeostasis
To counteract runaway growth, neuroscientists traditionally pointed to homeostatic plasticity—regulatory mechanisms like synaptic scaling that adjust a neuron's overall synaptic weight to maintain a target firing rate. However, relying solely on classical homeostasis introduces what Zenke terms the "temporal paradox" of neural learning.
Biologically, classical homeostatic scaling is a slow process, operating over hours or days. Yet, mathematical models demonstrate that Hebbian runaway excitation destroys network functionality within seconds or minutes. Slow homeostasis arrives far too late to save the network from saturation.
[Hebbian Co-Firing] ---> [Synaptic Strengthening] ---> [Higher Firing Rates] ---\
^ |
\----------------------- (Seconds / Minutes) ---------------------------/
[Homeostatic Scaling] --X--> [Slow Weight Reduction] (Hours / Days: Too Late!)
To resolve this paradox, the brain must deploy rapid compensatory processes that operate on the same timescale as Hebbian induction. Zenke et al. demonstrated that stable learning requires a homeostatic "thermostat" equipped with a fast rate detector operating on a timescale of seconds. On an even faster scale, Tegnér et al. proposed that postsynaptic calcium levels can dynamically tune the ratio of synaptic weakening to strengthening on a millisecond timescale, providing immediate adaptive control over STDP based on local neuronal dynamics.
Multiplicative Gain and Presynaptic Inhibition
Why can't standard, postsynaptic inhibition solve the runaway problem? The issue lies in the mathematical nature of the interaction. Postsynaptic inhibition typically acts subtractively on a neuron's inputs, lowering the membrane potential. However, Hebbian weight updates are inherently multiplicative because they depend on the product of pre- and postsynaptic activity. Subtractive inhibition is often insufficient to quench an explosive, multiplicative feedback loop.
To solve this, Naumann et al. demonstrated that presynaptic inhibition acts as a rapid, multiplicative gain control. By suppressing neurotransmitter release at the axon terminal before it ever reaches the postsynaptic site, presynaptic inhibition directly scales down the driving force of the Hebbian loop. This rapid gain control stabilizes recurrent excitation immediately, buying time for slower, permanent homeostatic mechanisms to consolidate the changes safely.
Anti-Hebbian Dynamics and Rate Derivatives
Another strategy involves altering the mathematical structure of the learning rule itself to include self-stabilizing terms. Rather than purely strengthening correlated connections, certain biological circuits incorporate anti-Hebbian components that penalize redundant or excessive activity.
For instance, Xie et al. showed that a spike-based learning rule depending on the derivative of a neuron's firing rate mathematically simplifies to an anti-Hebbian rule that naturally counteracts drift. This precise calibration prevents persistent activity from either decaying to zero or saturating to maximum, allowing working memory circuits to maintain stable representations without manual parameter tuning.
On an intracellular level, Bell proposed applying anti-Hebbian rules to the distribution of internal ion channels. By minimizing local voltage curvature, individual neurons can self-organize their internal conductance to prevent the formation of localized activity "hot-spots," ensuring stable and faithful signal conduction across the cell body.
Orchestrating Coordinated Plasticity
Ultimately, maintaining stability while preserving the capacity to learn requires an orchestration of multiple, interacting mechanisms. Zenke et al. demonstrated that spiking neural networks can stably form and retrieve associative memories by combining heterosynaptic plasticity (which weakens non-active pathways when active ones strengthen), homeostatic scaling, and synaptic consolidation rules.
When these local rules are properly balanced, networks can self-organize into a "balanced state" where excitatory and inhibitory inputs dynamically cancel each other out, as explored by Abbott et al. and Sadeh et al.. In this balanced regime, the network avoids runaway excitation while remaining highly sensitive to external stimuli, turning an inherently unstable feedback loop into a robust engine for distributed computation.
Key Foundational Work
Mathematical Analysis of Learning Behavior of Neuronal Models (1987) — contributed 2 concept elements to this direction.