Abstract
Characterizing the relation between weight structure and input/output statistics is fundamental for understanding the computational capabilities of neural circuits. In this work, I study the problem of storing associations between analog signals in the presence of correlations, using methods from statistical mechanics. I characterize the typical learning performance in terms of the power spectrum of random input and output processes. I show that optimal synaptic weight configurations reach a capacity of 0.5 for any fraction of excitatory to inhibitory weights and have a peculiar synaptic distribution with a finite fraction of silent synapses. I further provide a link between typical learning performance and principal components analysis in single cases. These results may shed light on the synaptic profile of brain circuits, such as cerebellar structures, that are thought to engage in processing time-dependent signals and performing on-line prediction.
Introduction
Biological neural networks are governed by fundamental constraints that distinguish them from typical machine learning architectures. One of the most prominent is Dale’s Law, which states that a neuron is either excitatory or inhibitory, meaning its synaptic weights must maintain a fixed sign. Furthermore, biological signals—such as those processed in the cerebellum—are rarely independent and identically distributed (i.i.d.); they exhibit significant spatio-temporal correlations. This paper investigates how these biological realities—sign constraints and signal correlations—impact a neuron's ability to store and associate analog patterns.
The Sign-Constrained Perceptron Model
To analyze this problem, we employ a linear perceptron model where the synaptic weights $w$ are partitioned into excitatory ($w > 0$) and inhibitory ($w < 0$) sets. Unlike standard models where weights can freely flip signs, this model enforces a fixed ratio of excitatory to inhibitory synapses. The goal is to find the "typical" learning behavior of such a system as it attempts to associate correlated input signals $X$ with a target output $y$.
Using the replica method from statistical mechanics, we derive the "free energy" of the learning process, treating synaptic weights as particles in a system minimizing an error function. This mean-field approach allows us to describe the system's behavior in the thermodynamic limit ($N \to \infty$), accounting for temporal correlations through Toeplitz covariance matrices that represent stationary discrete-time processes.
Universal Capacity and the Silent Synapse Phenomenon
A central finding of this research is the 0.5 Capacity Rule. In an unconstrained perceptron, the storage capacity is 1.0. However, the introduction of sign constraints (Dale's Law) reduces this capacity to exactly $\alpha_c = 0.5$. Remarkably, this value is universal: it remains constant regardless of the specific ratio of excitatory to inhibitory neurons or the underlying correlation structure of the data, provided the neuron's bias (threshold) is optimized.
Optimization of the bias leads to another striking biological parallel: the emergence of "silent" synapses. The theoretical derivation shows that in an optimal learning regime, exactly 50% of the synapses are set to zero weight. This matches experimental observations in structures like the cerebellum, where a large fraction of synapses appear non-functional or silent.
The Impact of Spatio-Temporal Correlations
Beyond capacity, the study establishes a direct mathematical link between the average minimal learning error and the power spectrum of the input and output signals. Sign constraints act as a form of implicit regularization, where the system's performance becomes sensitive to the principal components (PCA) of the input data. As signals become more confined to a lower-dimensional subspace (i.e., higher correlation), the learning performance degrades, highlighting the importance of signal dimensionality in neural computation.
Conclusion
This work provides a theoretical framework for understanding how E-I separated circuits process correlated analog signals. The discovery of a universal 0.5 capacity and the natural emergence of silent synapses suggests that these biological features may be optimized for information storage. Future work will extend this framework to non-linear activation functions and dynamic learning rules, such as Spike-Timing-Dependent Plasticity (STDP), to further bridge the gap between statistical mechanics and biological reality.
Summary updated