In traditional deep learning, networks learn by propagating a global error signal backward through layers of static connections. In contrast, biological neural networks reorganize themselves continuously using local rules. The foundational principle governing this self-organization is Hebbian plasticity, colloquially summarized as "neurons that fire together, wire together."
Rather than relying on an external supervisor to dictate correct behavior, a Hebbian learning rule instructs individual synapses to adjust their strengths based purely on the statistical correlation between pre-synaptic and post-synaptic activity. This local mechanism allows networks to autonomously discover the underlying structure of their environment.
Translating Correlation into Structure
Why does a simple correlation rule lead to powerful representation learning? At its core, a basic Hebbian rule acts as an online statistical engine. When a pre-synaptic neuron and a post-synaptic neuron fire simultaneously, the weight of the connection between them increases. Mathematically, this means the weight update is proportional to the product of input and output activity. As shown by Jia Liu et al. in Modeling Hebb Learning Rule for Unsupervised Learning, integrating this principle directly into an objective function allows a network to maximize the correlation between input vectors and hidden representations, extracting high-quality abstract features without needing a reconstruction task.
By reinforcing pathways that are frequently co-active, the network naturally aligns its physical connectivity with the statistical regularities of the outside world. For example, Martin I. Sereno demonstrated in Learning the Solution to the Aperture Problem for Pattern Motion with a Hebb Rule and Learning to See Rotation and Dilation with a Hebb Rule that stacking layers driven by unsupervised, correlation-based updates allows complex, position-independent feature detectors (such as pattern-selective motion units) to emerge spontaneously.
Standard Hebbian: Δw ∝ (Pre-synaptic Activity) × (Post-synaptic Activity)
Covariance Rule: Δw ∝ (Pre - Mean_pre) × (Post - Mean_post)
To capture more nuanced relationships than raw coincidences, networks often employ a covariance-based Hebbian rule, which updates weights based on the deviation of firing rates from their long-term means. As Kristofer E. Bouchard et al. explored in Role of the site of synaptic competition and the balance of learning forces for Hebbian encoding of probabilistic Markov sequences, this variation allows synapses to accurately encode complex, intermediate probability distributions, such as the Markovian transition probabilities underlying sequential behaviors.
The Emergence of Receptive Fields and Maps
One of the most remarkable properties of Hebbian learning is its ability to generate global order—such as topographic maps and specialized receptive fields—from purely localized interactions.
In sensory systems, natural stimuli are highly structured; for instance, visual scenes contain continuous edges and smooth motion trajectories. When a network is exposed to these regularities, a Hebbian rule selectively stabilizes connections between units that share functional properties. Carlos S. N. Brito demonstrated in Nonlinear Hebbian Learning as a Unifying Principle in Receptive Field Formation that diverse sensory development models (including sparse coding and Independent Component Analysis) can be mathematically unified under nonlinear Hebbian learning. This reveals that top-down optimization goals and bottom-up synaptic mechanisms are often equivalent descriptions of the same process: the network maximizing its efficiency by mirroring the statistical kurtosis of natural inputs.
This structural refinement operates even in the complete absence of external sensory experience or pre-existing spatial maps:
- Prenatal Wiring via Spontaneous Activity: Before an animal opens its eyes, waves of spontaneous electrical activity sweep across the retina. Jinwoo Kim et al. showed in Spontaneous Retinal Waves Can Generate Long-Range Horizontal Connectivity in Visual Cortex that a covariance-based learning rule leverages these internal signals to wire up long-range horizontal connections between neurons with similar orientation preferences.
- Self-Organization Without Topography: In sensory systems that lack organized spatial maps, such as the olfactory cortex, Julien Grimaud et al. found in Bilateral Alignment of Receptive Fields in the Olfactory Cortex that experience-driven Hebbian plasticity is sufficient to align disparate sensory inputs (like left and right nostrils) into a coordinated, unified representation.
- Pruning and Boundary Discovery: By combining lateral excitation with threshold-based pruning, networks can discover geometric regularities directly from raw data. In A Self-Organizing Neural Network for Contour Integration through Synchronized Firing, Yoonsuck Choe et al. demonstrated how dense initial connectivity refines itself into precise, trajectory-aligned structures by strengthening links between co-active units and eliminating weak, noisy connections.
Associative Memory and Cell Assemblies
Beyond wiring sensory filters, Hebbian learning provides a natural mechanism for memory storage through the creation of "cell assemblies"—localized clusters of neurons that exhibit synchronized firing patterns through learned associations (Ivana Kajic).
When information is stored in a fully connected network, the synaptic weights are configured based on the correlation between units within the desired memory patterns. As Wulfram Gerstner explained in Associative Memory in a Network of 'Biological' Neurons, this representational strategy ensures that the stored patterns correspond to stable fixed points in the network's state space. When the network is later exposed to a partial or noisy cue, its internal dynamics naturally gravitate toward these fixed points, achieving robust attractor-based retrieval. This architecture is highly effective for capturing the inherent biases and co-occurrence patterns found in highly structured data like natural language (Nigel Collier).
However, storing memories via correlational rules introduces a fundamental challenge: the updates inherently create strong statistical dependencies between different synapses. Cristina Savin et al. demonstrated in Correlations strike back (again): the case of associative memory retrieval that if a network assumes synaptic independence during recall, retrieval fails completely. To successfully navigate these dependencies, the retrieval dynamics must be "correlation-aware," utilizing biological features like feedback inhibition and non-linear dendritic processing to align the recall process with the structured organization of the weights.
Integrating Plasticity into Artificial Networks
In modern artificial intelligence, Hebbian mechanisms are increasingly used to bridge the gap between static, task-specific models and dynamic, adaptable systems. Rather than viewing local plasticity and global optimization as competing paradigms, recent architectures combine them to achieve rapid, on-the-fly adaptation.
Global Task Feedback (Gradient Descent)
│
▼
┌─────────────────────────────────────────┐
│ Neuromodulatory Control │
└────────────────────┬────────────────────┘
│ Controls learning rate
▼
┌─────────────────────────────────────────┐
│ Local Synaptic Plasticity (Hebbian) │
└─────────────────────────────────────────┘
For instance, Yinan Cao et al. mapped a continuous landscape of learning rules in Characterizing emergent representations in a space of candidate learning rules for deep networks, demonstrating that integrating unsupervised Hebbian components into supervised objectives allows networks to balance local correlations with global error signals, replicating human-like semantic development hierarchies.
This hybrid approach enables networks to update their own connectivity dynamically during inference:
- Rapid Few-Shot Learning: In Hebbian and Gradient-based Plasticity Enables Robust Memory and Rapid Learning in RNNs, Yu Duan et al. introduced Recurrent Neural Networks that utilize a neuromodulated Hebbian rule alongside gradient-based updates. This allows the network to form rapid associations between novel inputs during a single forward pass without requiring global backpropagation.
- Local Subspace Extraction: Instead of relying on computationally expensive global matrix decompositions to manage continual learning, Di He showed in Hebbian Learning based Orthogonal Projection for Continual Learning of Spiking Neural Networks that recurrent lateral connections using opposing Hebbian and anti-Hebbian rules can autonomously identify the principal subspace of activity patterns through purely local neuronal interactions.
By shifting the burden of fine-grained, associative adaptation to local Hebbian rules, artificial systems can achieve the flexibility, energy efficiency, and structural robustness characteristic of biological brains (Feng Xia et al.).
Key Foundational Work
A Computer Simulation of Olfactory Cortex with Functional Implications for Storage and Retrieval of Olfactory Information (1987) — contributed 1 concept elements to this direction.
Learning the Solution to the Aperture Problem for Pattern Motion with a Hebb Rule (1988) — contributed 1 concept elements to this direction.