Abstract Cortical populations of neurons develop sparse representations adapted to the statistics of the environment. While existing synaptic plasticity models reproduce some of the observed receptive-field properties, a major obstacle is the sensitivity of Hebbian learning to omnipresent spurious correlations in cortical networks which can overshadow relevant latent input features. Here we develop a theory for synaptic plasticity that is invariant to second-order correlations in the input. Going beyond classical Hebbian learning, we show how Hebbian long-term depression (LTD) cancels the sensitivity to second-order correlations, so that receptive fields become aligned with features hidden in higher-order statistics. Our simulations demonstrate how correlation-invariance enables biologically realistic models to develop sparse population codes, despite diverse levels of variability and heterogeneity. The theory advances our understanding of local unsupervised learning in cortical circuits and assigns a specific functional role to synaptic LTD mechanisms in pyramidal neurons.
Introduction
In the quest to understand how the brain learns, "sparse coding" is a leading theory. It suggests that neurons adapt to represent the environment using as few active cells as possible, which is incredibly efficient. However, a major technical hurdle exists: standard Hebbian learning ("neurons that fire together, wire together") is easily distracted by "spurious" correlations—loud, common signals that do not necessarily carry relevant information.
Much like Principal Component Analysis (PCA), standard Hebbian rules tend to latch onto the strongest second-order correlations rather than the rare, higher-order features that define sparse codes. In artificial intelligence, this is typically solved by a "whitening" pre-processing step. This paper explores how biological neurons might achieve similar correlation-invariance locally at the synapse.
The Theory of Correlation-Invariant Plasticity
The authors propose a synaptic plasticity rule derived from a "normalized projection pursuit" objective. The goal is for a neuron to maximize the sparseness of its output while maintaining a normalized response. The resulting weight change ($\Delta w$) is defined by:
$$\Delta w = \eta (x y^2 - h_y x y)$$
This rule consists of three critical components:
- Nonlinear Hebbian LTP ($x y^2$): This term drives the neuron to seek out non-Gaussian, sparse features.
- Linear Hebbian LTD ($-h_y x y$): This "anti-PCA" term is the core innovation. By being linear in both pre- and post-synaptic activity, it specifically cancels out the influence of second-order correlations.
- Metaplasticity ($h_y$): A homeostatic factor that tracks the neuron's average activity to ensure stability.
Key Insights
The proposed rule suggests that Long-Term Depression (LTD) is not merely a homeostatic "brake" to prevent runaway excitation. Instead, it serves a specific functional role: filtering out "boring" correlations so that the neuron can focus on interesting, hidden features. This mechanism allows a neuron to perform Independent Component Analysis (ICA) on raw data without external whitening. Furthermore, the rule naturally accounts for dendritic compensation, automatically boosting weights for distant synapses to maintain sensitivity to information rather than raw signal amplitude.
Results and Performance
The theory was validated across several simulated environments, including synthetic sparse signals, orientation tuning tasks, and natural image datasets.
- Signal Extraction in High Noise: In simulations where Gaussian noise was 20% stronger than the sparse signal, the correlation-invariant rule successfully identified the signal. In contrast, standard models like Oja’s rule failed, learning only the noise.
- Optimal Decoding and SNR: The model effectively weighted inputs based on their Signal-to-Noise Ratio (SNR), prioritizing reliable synapses. Performance metrics show that the output noise level achieved by the Hebb-LTD rule closely matches that of a mathematically optimal linear decoder, significantly outperforming standard rules.
- Emergence of V1 Receptive Fields: When applied to spiking networks processing natural images, the model developed localized, orientation-selective receptive fields. Notably, it achieved this without the lateral inhibition or pre-whitening steps required by previous models.
Conclusion and Future Directions
This research advances our understanding of local unsupervised learning in the brain. By assigning a functional role to LTD in pyramidal neurons, it explains how cortical circuits can remain robust to diverse levels of variability. Future work will need to address how neurons handle non-zero mean inputs and whether biological homeostatic mechanisms are fast enough to support this rule in real-time. Additionally, scaling these principles to deep, hierarchical networks remains a key challenge for future investigation.
Summary updated