Information-Theoretic Intrinsic Plasticity for Online Unsupervised Learning in Spiking Neural Networks

Wenrui Zhang, Peng LiFrontiers in Neuroscience 2019 · DOI

Introduction

Intrinsic plasticity (IP) is a fundamental homeostatic mechanism in biological neural circuits that regulates excitability to maintain stable firing rates and maximize information transmission. In the context of Spiking Neural Networks (SNNs), implementing effective IP rules has been challenging due to the non-differentiable nature of discrete spike events. Previous heuristic approaches often lacked theoretical rigor and failed to demonstrate significant performance gains in complex, real-world tasks. This paper introduces SpiKL-IP, a novel IP rule derived from information-theoretic principles, designed to optimize the firing rate distribution of spiking neurons for enhanced unsupervised learning.

The SpiKL-IP Framework

The core objective of SpiKL-IP is to maximize the entropy of a neuron's output firing rate distribution. According to information theory, for a fixed mean firing rate, an exponential distribution achieves maximum entropy. SpiKL-IP achieves this by minimizing the Kullback-Leibler (KL) divergence between the actual output distribution and a target exponential distribution.

Firing-Rate Transfer Function (FR-TF)

To enable gradient-based optimization in SNNs, the authors derived a differentiable Firing-Rate Transfer Function (FR-TF) for the Leaky Integrate-and-Fire (LIF) neuron model. This function relates the input current to the steady-state firing rate.

The firing-rate transfer function (FR-TF) as a function of leaky resistance R (A) and membrane time constant τm (B).

As shown in Figure 1, adjusting the leaky resistance ($R$) and the membrane time constant ($τ_m$) allows for precise control over the neuron's gain and threshold. SpiKL-IP adaptively tunes these parameters in an online fashion, allowing neurons to self-organize their internal dynamics based on the statistics of their input.

Distribution Shaping

The fundamental operation of SpiKL-IP is the transformation of input statistics into a high-entropy output. By adjusting internal parameters, the neuron acts as a non-linear filter that shapes the probability density of its firing rate.

Transformation of input distribution to an optimal exponential output distribution through a spiking neuron.

Figure 2 illustrates how a neuron, governed by SpiKL-IP, can take a Gaussian-like input distribution and shape its output into the desired exponential form. This process ensures that the neuron utilizes its full dynamic range efficiently, maximizing the information content of its spike trains.

Experimental Evaluation

The effectiveness of SpiKL-IP was evaluated using Liquid State Machines (LSMs) on two distinct benchmarks: the TI46 speech corpus and the CityScape image dataset.

Speech Recognition

On the challenging TI46 multi-speaker task, which involves recognizing spoken English letters from multiple individuals, SpiKL-IP demonstrated a remarkable performance boost. Compared to a baseline LSM without IP, the proposed rule increased classification accuracy by over 16%. It also significantly outperformed existing heuristic IP methods, such as Voltage-Threshold IP and RC IP.

Image Classification

In semantic urban scene classification using the CityScape dataset, SpiKL-IP consistently improved accuracy across various reservoir sizes. For instance, in a mid-sized network, accuracy rose from 91.7% to 94.4%. These results highlight the rule's ability to extract meaningful features from complex spatial data in an unsupervised manner.

Discussion and Future Work

SpiKL-IP represents a significant step forward in bridging the gap between biological realism and computational efficiency in SNNs. By moving from ad hoc heuristics to a rigorous information-theoretic foundation, the authors have provided a robust tool for online unsupervised learning.

Key insights from this work include:

  • Homeostasis as Information Optimization: IP is not merely about rate stabilization but about shaping activity distributions to maximize throughput.
  • Parameter Synergy: Tuning both $R$ and $τ_m$ provides superior control over neural dynamics compared to threshold-only adjustments.

Future research directions include integrating SpiKL-IP with synaptic plasticity rules like Spike-Timing-Dependent Plasticity (STDP) to explore the synergy between intrinsic and synaptic learning. Additionally, testing the rule's robustness in stochastic neuron models and scaling it to deep, multi-layer SNN architectures remain important frontiers.

Summary updated

Abstract

As a self-adaptive mechanism, intrinsic plasticity (IP) plays an essential role in maintaining homeostasis and shaping the dynamics of neural circuits. From a computational point of view, IP has the potential to enable promising non-Hebbian learning in artificial neural networks. While IP based learning has been attempted for spiking neuron models, the existing IP rules are <i>ad hoc</i> in nature, and the practical success of their application has not been demonstrated particularly toward enabling real-life learning tasks. This work aims to address the theoretical and practical limitations of the existing works by proposing a new IP rule named SpiKL-IP. SpiKL-IP is developed based on a rigorous information-theoretic approach where the target of IP tuning is to maximize the entropy of the output firing rate distribution of each spiking neuron. This goal is achieved by tuning the output firing rate distribution toward a targeted optimal exponential distribution. Operating on a proposed firing-rate transfer function, SpiKL-IP adapts the intrinsic parameters of a spiking neuron while minimizing the KL-divergence from the targeted exponential distribution to the actual output firing rate distribution. SpiKL-IP can robustly operate in an online manner under complex inputs and network settings. Simulation studies demonstrate that the application of SpiKL-IP to individual neurons in isolation or as part of a larger spiking neural network robustly produces the desired exponential distribution. The evaluation of SpiKL-IP under real-world speech and image classification tasks shows that SpiKL-IP noticeably outperforms two existing IP rules and can significantly boost recognition accuracy by up to more than 16%.
BibTeX
@inproceedings{zhang2019information,
  author = {Wenrui Zhang and Peng Li},
  title = {{Information-Theoretic Intrinsic Plasticity for Online Unsupervised Learning in Spiking Neural Networks}},
  booktitle = {Frontiers in Neuroscience},
  year = {2019},
  doi = {10.3389/fnins.2019.00031},
}
Figure