Liquid State Machine with Dendritically Enhanced Readout for Low-power, Neuromorphic VLSI Implementations

Subhrajit Roy, Amitava Banerjee, Arindam Basu

Abstract

In this paper, we describe a new neuro-inspired, hardware-friendly readout stage for the liquid state machine (LSM), a popular model for reservoir computing. Compared to the parallel perceptron architecture trained by the p-delta algorithm, which is the state of the art in terms of performance of readout stages, our readout architecture and learning algorithm can attain better performance with significantly less synaptic resources making it attractive for VLSI implementation. Inspired by the nonlinear properties of dendrites in biological neurons, our readout stage incorporates neurons having multiple dendrites with a lumped nonlinearity. The number of synaptic connections on each branch is significantly lower than the total number of connections from the liquid neurons and the learning algorithm tries to find the best 'combination' of input connections on each branch to reduce the error. Hence, the learning involves network rewiring (NRW) of the readout network similar to structural plasticity observed in its biological counterparts. We show that compared to a single perceptron using analog weights, this architecture for the readout can attain, even by using the same number of binary valued synapses, up to 3.3 times less error for a two-class spike train classification problem and 2.4 times less error for an input rate approximation task. Even with 60 times larger synapses, a group of 60 parallel perceptrons cannot attain the performance of the proposed dendritically enhanced readout. An additional advantage of this method for hardware implementations is that the 'choice' of connectivity can be easily implemented exploiting address event representation (AER) protocols commonly used in current neuromorphic systems where the connection matrix is stored in memory. Also, due to the use of binary synapses, our proposed method is more robust against statistical variations.

What Problem Does This Paper Address?

Liquid State Machines (LSMs) are a popular form of "reservoir computing" where an input is projected into a high-dimensional, recurrent "liquid" of neurons. While the liquid itself doesn't need training, the readout stage typically does.

Overview of a Liquid State Machine (LSM) architecture showing the input, liquid reservoir, and readout stage.

Standard readouts use a layer of parallel perceptrons (LSM-PPR) that require a massive number of high-resolution, tunable synaptic weights. In the world of Very Large Scale Integration (VLSI) circuits, these weights are expensive: they consume significant silicon area, require complex memory structures, and drain power. Furthermore, analog hardware is prone to "mismatch" (statistical variations in components), which can degrade the performance of traditional learning algorithms like $p$-delta.

Methods

The authors propose a Dendritically Enhanced Readout (LSM-DER). Instead of a simple linear summation of inputs, they use a two-compartment neuron model inspired by biological structural plasticity.

The proposed Dendritically Enhanced Readout (DER) stage featuring multiple dendritic branches per neuron and binary synapses.
  1. Nonlinear Dendrites: Readout neurons have multiple dendritic branches. Each branch performs a local summation of inputs followed by a lumped nonlinearity (a square-law function with saturation).
  2. Binary Synapses: To make the system hardware-friendly, the synapses are binary (on/off) rather than high-precision analog values.
  3. Network Rewiring (NRW): Since the weights are binary, "learning" doesn't involve changing weight values. Instead, the algorithm performs structural plasticity—it logically re-routes connections. It identifies poorly performing synapses and replaces them with new connections from a random pool, guided by a gradient-descent-based performance index.

Datasets

The architecture was evaluated on two standard benchmark tasks for LSMs:

  • Task I: Spike Train Classification: Distinguishing between two classes of Poisson spike trains. The test set included "jittered" versions of the templates to test generalization.
  • Task II: Retrieval of Sum of Rates: A regression task where the network must output the sum of firing rates of multiple input streams averaged over a past time window.

Key Insights and Ideas

  • Structural vs. Synaptic Plasticity: For hardware with limited precision, changing who you talk to (connectivity) is more powerful than changing how loudly you talk (weight).
  • Dendritic Gain: By adding nonlinearities at the dendritic level before the final soma (cell body) summation, the "storage capacity" of a single neuron increases exponentially. This allows a single DER neuron to outperform a large group of standard perceptrons.
  • Hardware Robustness: Because the system relies on the combination of many binary connections rather than the precise value of a few analog ones, it is inherently more robust to manufacturing variations common in sub-threshold CMOS circuits.

Key Results

  • Superior Accuracy: Compared to the state-of-the-art $p$-delta algorithm using the same number of synapses, the LSM-DER achieved 3.3x less error in spike train classification and 2.4x less error in the rate approximation task.
  • Resource Efficiency: Even when a group of 60 parallel perceptrons used 60 times more synapses, they still could not match the performance of the proposed dendritically enhanced readout.
  • Silicon Area Savings: In a simulated VLSI implementation, the DER readout was approximately 4.67 times smaller in area than a comparable PPR readout.
  • Robustness: Monte Carlo simulations showed that when hardware non-idealities were introduced, the error for LSM-DER increased by only ~0.02, whereas the standard PPR error increased by ~0.047.

Conclusion

The LSM-DER architecture offers a significant leap in efficiency for neuromorphic hardware. By mimicking the nonlinear processing and structural plasticity of biological dendrites, it achieves high performance with low-precision, binary synapses. Future work will focus on implementing the rewiring process directly on-chip to create fully autonomous, low-power learning systems.

Summary updated

Abstract

In this paper, we describe a new neuro-inspired, hardware-friendly readout stage for the liquid state machine (LSM), a popular model for reservoir computing. Compared to the parallel perceptron architecture trained by the p-delta algorithm, which is the state of the art in terms of performance of readout stages, our readout architecture and learning algorithm can attain better performance with significantly less synaptic resources making it attractive for VLSI implementation. Inspired by the nonlinear properties of dendrites in biological neurons, our readout stage incorporates neurons having multiple dendrites with a lumped nonlinearity. The number of synaptic connections on each branch is significantly lower than the total number of connections from the liquid neurons and the learning algorithm tries to find the best 'combination' of input connections on each branch to reduce the error. Hence, the learning involves network rewiring (NRW) of the readout network similar to structural plasticity observed in its biological counterparts. We show that compared to a single perceptron using analog weights, this architecture for the readout can attain, even by using the same number of binary valued synapses, up to 3.3 times less error for a two-class spike train classification problem and 2.4 times less error for an input rate approximation task. Even with 60 times larger synapses, a group of 60 parallel perceptrons cannot attain the performance of the proposed dendritically enhanced readout. An additional advantage of this method for hardware implementations is that the 'choice' of connectivity can be easily implemented exploiting address event representation (AER) protocols commonly used in current neuromorphic systems where the connection matrix is stored in memory. Also, due to the use of binary synapses, our proposed method is more robust against statistical variations.
BibTeX
@article{roy2014liquid,
  author = {Subhrajit Roy and Amitava Banerjee and Arindam Basu},
  title = {{Liquid State Machine with Dendritically Enhanced Readout for Low-power, Neuromorphic VLSI Implementations}},
  journal = {IEEE Trans. Biomed. Circuits Syst.},
  year = {2014},
  url = {https://openreview.net/forum?id=uSOaWhKXyz},
}
Figure