Biologically plausible solutions for spiking networks with efficient coding

Veronika Koren, Stefano Panzeri

Introduction

Understanding how the dynamics of neural networks are shaped by the computations they perform is a fundamental question in neuroscience. Researchers have recently derived a spiking neural network model that is both mathematically optimal for "efficient coding" and biologically realistic. By starting from first principles of error minimization, the authors demonstrate that key biological features—like Dale’s Law, fast-spiking inhibitory neurons, and adaptation currents—are not just biological "quirks" but necessary mathematical solutions for efficient information processing.

The Tension: Efficiency vs. Biology

In neuroscience, there is a long-standing tension between two worlds: Efficient Coding Theory and Biological Realism.

Efficient coding suggests that the brain’s goal is to represent information using as few spikes (and as little energy) as possible. Mathematical models of this theory are elegant but often ignore the "messy" reality of biology. For example, many models allow a single neuron to be both excitatory and inhibitory, or they use unrealistic "all-to-all" connectivity.

Conversely, biological models (like the Leaky Integrate-and-Fire model) are realistic but often lack a clear functional "why"—we know how the neurons fire, but not necessarily what global objective they are optimizing. This paper seeks to bridge that gap by deriving a network that satisfies biological constraints while remaining mathematically optimal for tracking a signal.

Methods: Deriving Biology from First Principles

The authors utilize a Loss Function approach. They hypothesize that the network aims to minimize two primary objectives:

  1. Coding Error: The difference between the actual stimulus and the network's internal estimate.
  2. Metabolic Cost: A penalty for firing too many spikes.

The core novelty lies in how they split this objective. They define separate loss functions for excitatory (E) and inhibitory (I) populations. By analytically deriving the conditions under which a neuron should fire to decrease these losses, they naturally "discover" the equations for a generalized Leaky Integrate-and-Fire (LIF) network.

Network architecture and emergent dynamics in the efficient coding model. Panel A shows the E-I network structure, while Panels B and C demonstrate signal tracking and emergent oscillatory behavior.

Key technical features of this derived network include:

  • Dale’s Law Enforcement: They mathematically ensure neurons are strictly either excitatory or inhibitory.
  • Low-Rank Connectivity: Synaptic weights are structured based on the similarity of what neurons encode, rather than being random.
  • Analytical Derivation of Currents: Instead of adding biological currents (like adaptation) by hand, these currents emerge naturally from the math of the loss function.

(Note: As a theoretical study, the authors validated their model using synthetic, low-dimensional stimulus signals rather than standard external datasets like MNIST, ensuring the network could accurately track and transform input features in real-time.)

Key Insights: Why the Brain Works the Way It Does

  • Biology as an Optimal Solution: The paper suggests that features observed in real brains—like inhibitory neurons having faster membrane time constants than excitatory ones—are actual requirements for efficient coding. The math dictates that I-neurons must be faster to keep the E-neurons in check.
  • The "Why" of Adaptation: Spike-triggered adaptation (where a neuron slows down its firing over time) is shown to be the physical implementation of the "metabolic cost" term in the loss function.
  • Rebound Currents (h-currents): The model predicts a depolarizing current after a spike, which perfectly mirrors the biological "rebound" current used to generate oscillations in the cortex.
  • Connectivity Complexity: The authors show that the "rank" (complexity) of the E-E connectivity matrix determines the network's behavior. Low-complexity networks simply follow the input, while high-complexity networks can generate their own internal dynamics.

Key Results: Balance, Oscillations, and Spontaneous Activity

  • Functional E-I Balance: The derived network successfully performs linear transformations of input signals while maintaining a biologically plausible balance of excitation and inhibition.
  • Emergent Oscillations: When the network includes E-E connectivity and adaptation, it produces population-wide oscillations (rhythmic firing), a hallmark of real brain activity that simpler efficient coding models fail to replicate.
  • Spontaneous Activity: Even without an external stimulus, the network exhibits "Up" and "Down" states (periods of high activity followed by silence), driven by noise and amplified by recurrent excitatory connections.
  • Distributed Coding: In complex versions of the model, neurons that aren't directly "tuned" to a stimulus still participate in the representation, showing how information is distributed across the population.

Limitations and Future Directions

While a significant step forward, the current derivation is limited to linear transformations of inputs, whereas real-world neural computation is often highly non-linear. Additionally, the network's connectivity is "prescribed" by the math rather than learned over time. The authors suggest that their derived "facilitation" current might be the key to future work on how these networks could learn through synaptic plasticity. Finally, while the model is biologically plausible, future benchmarking is needed to quantify exactly how much "efficiency" is lost when moving from an abstract mathematical model to this more constrained biological reality.

Summary updated

Abstract

Understanding how the dynamics of neural networks is shaped by the computations they perform is a fundamental question in neuroscience. Recently, the framework of efficient coding proposed a theory of how spiking neural networks can compute low-dimensional stimulus signals with high efficiency. Efficient spiking networks are based on time-dependent minimization of a loss function related to information coding with spikes. To inform the understanding of the function and dynamics of biological networks in the brain, however, the mathematical models have to be informed by biology and obey the same constraints as biological networks. Currently, spiking network models of efficient coding have been extended to include some features of biological plausibility, such as architectures with excitatory and inhibitory neurons. However, biological realism of efficient coding theories is still limited to simple cases and does not include single neuron and network properties that are known to be key in biological circuits. Here, we revisit the theory of efficient coding with spikes to develop spiking neural networks that are closer to biological circuits. Namely, we find a biologically plausible spiking model realizing efficient coding in the case of a generalized leaky integrate-and-fire network with excitatory and inhibitory units, equipped with fast and slow synaptic currents, local homeostatic currents such as spike-triggered adaptation, hyperpolarization-activated rebound current, heterogeneous firing thresholds and resets, heterogeneous postsynaptic potentials, and structured, low-rank connectivity. We show how the rank of E-E connectivity matrix shapes network responses.
BibTeX
@inproceedings{koren2022biologically,
  author = {Veronika Koren and Stefano Panzeri},
  title = {{Biologically plausible solutions for spiking networks with efficient coding}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=zdmYnIRXvKS},
}
Figure