The computational and learning benefits of Daleian neural networks

Adam Haber, Elad SchneidmanNeurIPS 2022

This paper investigates a fundamental constraint of biological brains known as Dale’s Principle: the rule that a single neuron typically releases the same neurotransmitters at all its synapses, making it either strictly excitatory or strictly inhibitory.

While Dale’s Principle exponentially restricts the number of possible network architectures, this research demonstrates that "Daleian" networks can approximate the computations of unconstrained networks with remarkably high accuracy. Furthermore, these biological constraints actually confer functional benefits, including increased robustness to synaptic noise and more efficient learning through simple, neuron-wide parameter tuning.

The Problem: What is the Brain Missing?

In standard Artificial Neural Networks (ANNs), a single unit can have both positive (excitatory) and negative (inhibitory) outgoing weights. In biology, this is almost never the case. This "Daleian" constraint is mathematically massive: for a network of just 15 neurons, non-Daleian configurations outnumber Daleian ones by a factor of $10^{32}$.

Diagram illustrating non-Daleian vs. Daleian network architectures and their spiking responses to stimuli.

The authors ask a fundamental question: What is the brain "missing" by imposing this restriction? Does it sacrifice computational richness for biological simplicity, or is there a hidden functional advantage to being Daleian?

Methods and Datasets

To answer this, the researchers compared Daleian and non-Daleian recurrent neural networks using two distinct models:

  1. Spiking Neural Networks: Discrete-time models where activity is binary (spike or silence), and dynamics are treated as a Markov chain converging to a stationary distribution.
  2. Rate-based Networks: Continuous-time recurrent firing-rate models, which allow for mathematical tractability in larger networks.

To measure "functional similarity," they used the Jensen-Shannon divergence ($D_{JS}$) to compare the response distributions of different networks to the same stimuli. They also employed the Hessian of the Fisher Information Matrix to analyze the "flatness" of the functional landscape, indicating how sensitive a network is to small changes in its parameters.

The study utilized both synthetic and biological data. The authors generated ensembles of randomly connected networks (5,000 Daleian and 5,000 non-Daleian) and tested them with random stimuli. To validate their learning models, they also used real spiking patterns recorded from the prefrontal cortex of macaque monkeys performing a visual classification task, allowing them to see how well Daleian networks could approximate empirical biological responses.

Key Insights and Ideas

  • Functional Tiling: Surprisingly, Daleian networks are not "stuck" in a small corner of possible functions. They "tile" the space of neural response distributions almost as thoroughly as non-Daleian networks. For nearly any arbitrary non-Daleian network, there exists a Daleian "neighbor" that performs a nearly identical computation.
tSNE embedding showing the overlapping functional spaces of Daleian and non-Daleian networks.
  • Robustness to Synaptic Noise: Daleian networks are significantly more robust to random fluctuations in synaptic weights. In a biological environment where synapses are "noisy" and prone to spontaneous remodeling, this stability is a major evolutionary advantage.
  • The "Neuron-Level" Learning Advantage: In non-Daleian networks, learning usually requires tuning every individual synapse. In Daleian networks, the authors found that simply tuning three parameters per neuron (input scaling, output scaling, and baseline bias) allowed the network to learn nearly as effectively as if it were tuning every single synapse. This suggests a much simpler, more biologically plausible mechanism for learning.

Key Results

  • High-Accuracy Approximation: Optimized Daleian networks were able to approximate non-Daleian targets with an average dissimilarity ($D_{JS}$) of $\sim 5 \cdot 10^{-4}$ bits. This approximation is so accurate that it would take thousands of observations to tell the two networks apart.
  • Sensitivity Ratios: In 98% of tested stimuli, Daleian networks were less sensitive to synaptic noise than their non-Daleian counterparts. Conversely, they were more sensitive to changes in neuronal biases, making them perfectly "tuned" for neuron-level adaptation.
  • Information Capacity: Randomly sampled Daleian networks were found to be more informative about their stimuli than random non-Daleian networks, suggesting that the Daleian constraint naturally pushes networks toward more useful functional regimes.

Limitations and Future Work

The authors acknowledge several limitations in their current study. First, the research focused on relatively small ($N=10$ to $100$) and dense networks, whereas real biological circuits are often much larger and sparser. Second, while they tested different weight distributions, they did not deeply explore complex structural motifs (like "small-world" or "scale-free" properties) that are common in the brain.

Looking forward, the authors hypothesize that the robustness and neuron-level tuning of Daleian networks might help prevent catastrophic forgetting in AI systems. By showing that Dale's Principle is a "feature, not a bug," this paper provides a compelling argument for incorporating biological constraints into the design of more robust and efficient artificial intelligence.

Summary updated

BibTeX
@inproceedings{haber2022computational,
  author = {Adam Haber and Elad Schneidman},
  title = {{The computational and learning benefits of Daleian neural networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
}
Figure