Abstract
Previous theoretical studies on the interaction of excitatory and inhibitory neurons proposed to model this cortical microcircuit motif as a so-called Winner-Take-All (WTA) circuit. A recent modeling study however found that the WTA model is not adequate for data-based softer forms of divisive inhibition as found in a microcircuit motif in cortical layer 2/3. We investigate here through theoretical analysis the role of such softer divisive inhibition for the emergence of computational operations and neural codes under spike-timing dependent plasticity (STDP). We show that in contrast to WTA models - where the network activity has been interpreted as probabilistic inference in a generative mixture distribution - this network dynamics approximates inference in a noisy-OR-like generative model that explains the network input based on multiple hidden causes. Furthermore, we show that STDP optimizes the parameters of this model by approximating online the expectation maximization (EM) algorithm. This theoretical analysis corroborates a preceding modelling study which suggested that the learning dynamics of this layer 2/3 microcircuit motif extracts a specific modular representation of the input and thus performs blind source separation on the input statistics.
Introduction: Beyond Winner-Take-All
For decades, computational neuroscientists have modeled cortical circuits as Winner-Take-All (WTA) networks. In these models, strong lateral inhibition ensures that only one neuron (the "winner") fires in response to an input. Mathematically, this maps to "mixture models," where the brain assumes every sensory input has exactly one cause.
However, biological evidence from cortical layer 2/3 shows that inhibition is "softer" and "divisive." Instead of a single winner, multiple neurons often fire simultaneously to represent different features of a single scene. This research addresses two primary questions:
- What is the mathematical objective of a circuit that allows multiple winners?
- How does a local biological mechanism like Spike-Timing Dependent Plasticity (STDP) enable such a circuit to learn the structure of the world without global supervision?
Methods: Bridging Biology and Statistical Theory
The authors bridge the gap between biological realism and statistical theory using a multi-step approach:
- The Spiking Model: They utilize a data-based model of excitatory (E) and inhibitory (I) neurons. Unlike rigid WTA models, this one employs divisive inhibition, where total network activity scales with the input.
- The Probabilistic Model: They propose a generative model based on a noisy-OR likelihood. In this framework, an input (e.g., a pixel being "on") is explained if at least one of its potential hidden causes is active. This is paired with a Gaussian prior that encourages sparsity, resulting in an "adaptive k-WTA" behavior.
- The Theoretical Link: Using Neural Sampling Theory, the authors prove that the spiking dynamics of the network perform "inference"—specifically, neurons draw samples from the posterior distribution of hidden causes given the input.
- Learning as Expectation Maximization (EM): They derive a learning rule by taking the gradient of the model's log-likelihood. They demonstrate that this theoretical rule, which approximates the EM algorithm, can be simplified into a local, biologically plausible STDP rule.
Datasets and Evaluation
The model was evaluated using several benchmarks:
- Földiák’s Superposition-of-Bars: A classic benchmark for Blind Source Separation. The network is presented with 8x8 pixel grids containing random combinations of horizontal and vertical bars. The challenge is to "unmix" these and assign individual neurons to represent individual bars.
- Synthetic Poisson Spike Trains: To simulate realistic input, pixel intensities were converted into spike rates (e.g., 75 Hz for a black pixel, 1 Hz for white).
- Orientation Selectivity Data: The authors compared the model's output to experimental data from orientation "pinwheels" in the visual cortex.
Key Insights
- Modular Representation: The shift from WTA to k-WTA represents a transition from a "mixture model" to a "modular representation." This allows the brain to represent complex objects as combinations of simpler parts.
- STDP as an EM Engine: A critical finding is that local synaptic changes (STDP) are mathematically equivalent to an online version of the Expectation Maximization algorithm, suggesting the brain acts as an efficient statistical engine.
- "Explaining Away": Due to the noisy-OR structure, if one neuron "explains" a part of the input, it reduces the drive for other neurons to represent that same part. This competition enables the network to perform Blind Source Separation.
Key Results
The network successfully learned to disentangle superimposed lines in the bars task. Individual neurons became specialized for specific bars, even when presented in combination with others. Unlike WTA models, which produce unnaturally sparse codes, this model produced Gaussian-like tuning curves matching orientation selectivity observed in the living visual cortex.
The authors also measured the Kullback-Leibler (KL) divergence between their simplified spiking model and the ideal mathematical posterior. The divergence remained low (~0.55), proving the biological approximation is a strong fit for the underlying mathematics.
Limitations and Future Work
The mathematical derivation relies on symmetric recurrent weights, which is a simplification of biological reality. Additionally, the model uses simplified "rectangular" postsynaptic potentials (PSPs) for the core proof, though simulations with more realistic "double-exponential" PSPs remained robust. A major next step involves understanding how these noisy-OR modules stack into deep hierarchies to represent increasingly abstract concepts.
Summary updated