Optimal learning with excitatory and inhibitory synapses

Alessandro Ingrosso

Introduction

Biological neural networks operate under fundamental constraints that differ significantly from standard artificial neural networks. One of the most prominent is Dale’s Law, which states that a single neuron typically forms only one type of synapse—either excitatory or inhibitory—across all its axonal targets. While modern machine learning allows weights to fluctuate freely between positive and negative values, biological systems must learn within these rigid sign constraints.

This research, published in PLOS Computational Biology, investigates the theoretical limits of learning in single neurons constrained by Dale's Law. By applying tools from statistical mechanics to linear neural models, the study reveals how these biological constraints interact with the complex, correlated statistical structure of real-world data.

The Biological Constraint: Dale's Law and Correlated Data

In standard perceptron models, synapses are represented by weights that can be any real number. However, in the brain, an excitatory neuron only sends "go" signals, and an inhibitory one only sends "stop" signals. Previous theoretical studies using "Gardner-style" analysis often assumed that the data being learned was random and uncorrelated (i.e., independent and identically distributed, or i.i.d.).

In reality, the signals moving through the brain are highly structured and temporally correlated. This paper addresses a critical gap in computational neuroscience: how do biological sign constraints limit a neuron's learning capacity when faced with the complex statistical structure of naturalistic data?

Methodology: The Linear Perceptron and Replica Method

The author employs a linear perceptron model where weights are strictly non-negative for excitatory synapses and non-positive for inhibitory ones. To analyze the "typical" behavior of this system in the thermodynamic limit (where the number of neurons $N \to \infty$), the paper utilizes the replica method, a sophisticated technique derived from the physics of disordered systems, specifically spin glasses.

Schematic of the sign-constrained linear perceptron model with excitatory (red) and inhibitory (blue) inputs.

Key methodological components include:

  • Mean-Field Theory: Deriving equations that describe the system's state as $N$ becomes very large.
  • Covariance Modeling: Instead of simple white noise, the author uses Toeplitz matrices to model temporal correlations and the Stieltjes transform to analyze the eigenvalue spectrum of the input data.
  • Bias Optimization: A central finding is that treating the neuron's firing threshold (bias) as a learnable parameter is essential for achieving optimal performance under sign constraints.

To maintain mathematical rigor, the study uses synthetic statistical models rather than natural datasets. These include i.i.d. Gaussian data as a baseline, time-correlated signals generated with Gaussian (RBF) and exponential kernels, and orthogonal PCA models where the singular value spectrum can be manually controlled.

Key Insights: The Universal 0.5 Capacity

The research yields several profound insights into the nature of biological learning:

  1. The Universal 0.5 Capacity: In an unconstrained linear system, one can perfectly store $N$ patterns using $N$ weights, resulting in a capacity of 1.0. This study proves that adding sign constraints cuts this capacity exactly in half to 0.5. Remarkably, this 0.5 limit holds regardless of the ratio of excitatory to inhibitory connections, provided the neuron's bias is optimized.
  2. The Necessity of Silent Synapses: The mathematical derivation shows that at the limit of learning capacity, exactly 50% of a neuron's potential synapses should be "silent" (zero weight). This provides a strong theoretical justification for the prevalence of silent synapses observed in biological structures like the cerebellum.
  3. Performance is Spectral: The average learning error is a direct function of the power spectrum of the input signals. If the input data is highly redundant (low dimensionality), the neuron's ability to learn unique associations decreases significantly.

Results: Error and Weight Distributions

The paper provides a closed-form analytical expression that predicts the minimum possible regression error based solely on the eigenvalues of the input covariance matrix. Optimal weights are found to follow a distribution consisting of two truncated Gaussians (one for excitatory, one for inhibitory) and a massive "delta peak" at zero, representing the 50% silent synapses.

Numerical results showing (A) regression error vs. capacity for different noise levels and (B) convergence of model parameters with increasing N.

Numerical simulations perfectly match the theoretical predictions. As shown in the results, a neuron with 80% excitatory synapses performs identically to one with 50%, as long as the bias is appropriately tuned. This suggests that the brain may be robust to variations in E/I ratios through the adjustment of firing thresholds.

Discussion and Limitations

While this work provides a rigorous foundation for understanding sign-constrained learning, several avenues for future research remain:

  • Non-linearity: The current model is strictly linear. Real neurons possess non-linear activation functions (e.g., ReLU or sigmoid), which would likely alter the capacity and weight distributions.
  • Teacher-Student Framework: The analysis assumes input and output are statistically independent. Incorporating a framework where a "teacher" process generates the output would better reflect many machine learning contexts.
  • Learning Dynamics: The paper identifies the optimal weight configuration but does not specify the biologically plausible learning rules required to reach that state.
  • Network Complexity: The analysis is limited to a single perceptron. Extending these findings to multi-layer or recurrent networks remains a significant open challenge.

Summary updated

Abstract

Author summary A general analysis of learning with biological synaptic constraints in the presence of statistically structured signals is lacking. Here, analytical techniques from statistical mechanics are leveraged to analyze association storage between analog inputs and outputs with excitatory and inhibitory synaptic weights. The linear perceptron performance is characterized and a link is provided between the weight distribution and the correlations of input/output signals. This formalism can be used to predict the typical properties of perceptron solutions for single learning instances in terms of the principal component analysis of input and output data. This study provides a mean-field theory for sign-constrained regression of practical importance in neuroscience as well as in adaptive control applications.
BibTeX
@article{ingrosso2020optimal,
  author = {Alessandro Ingrosso},
  title = {{Optimal learning with excitatory and inhibitory synapses}},
  journal = {PLOS Computational Biology},
  year = {2020},
  doi = {10.1371/journal.pcbi.1008536},
}
Figure