Liquid State Machine With Dendritically Enhanced Readout for Low-Power, Neuromorphic VLSI Implementations

Subhrajit Roy, Amitava Banerjee, Arindam Basu

TL;DR

This paper introduces a hardware-efficient readout stage for Liquid State Machines (LSMs) that mimics the nonlinear processing of biological dendrites. By using "Network Rewiring" to optimize connections rather than tuning high-precision weights, the authors achieve significantly higher accuracy with far fewer synaptic resources, making it ideal for low-power, neuromorphic VLSI chips.

What Problem Does This Paper Address?

Liquid State Machines are a popular form of "reservoir computing" where an input is projected into a high-dimensional, recurrent "liquid" of neurons. While the liquid itself doesn't need training, the readout stage typically does.

Standard readouts use a layer of parallel perceptrons (LSM-PPR) that require a massive number of high-resolution, tunable synaptic weights. In the world of Very Large Scale Integration (VLSI) circuits, these weights are expensive: they consume significant silicon area, require complex memory structures, and drain power. Furthermore, analog hardware is prone to "mismatch" (statistical variations in components), which can degrade the performance of traditional learning algorithms like $p$-delta.

Methods

The authors propose a Dendritically Enhanced Readout (LSM-DER). Instead of a simple linear summation of inputs, they use a two-compartment neuron model:

  1. Nonlinear Dendrites: Readout neurons have multiple dendritic branches. Each branch performs a local summation of inputs followed by a lumped nonlinearity (a square-law function with saturation).
  2. Binary Synapses: To make the system hardware-friendly, the synapses are binary (on/off) rather than high-precision analog values.
  3. Network Rewiring (NRW): Since the weights are binary, "learning" doesn't involve changing weight values. Instead, the algorithm performs structural plasticity—it physically (or logically, via an address table) re-routes connections. It identifies poorly performing synapses and replaces them with new connections from a random pool, guided by a gradient-descent-based performance index.

Datasets

The architecture was evaluated on two standard benchmark tasks for LSMs:

  • Task I: Spike Train Classification: Distinguishing between two classes of Poisson spike trains. The test set included "jittered" versions of the templates to test generalization.
  • Task II: Retrieval of Sum of Rates: A regression/approximation task where the network must output the sum of firing rates of multiple input streams averaged over a past time window.

Key Insights and Ideas

  • Structural vs. Synaptic Plasticity: The "aha moment" here is that for hardware with limited precision, changing who you talk to (connectivity) is more powerful than changing how loudly you talk (weight).
  • Dendritic Gain: By adding nonlinearities at the dendritic level before the final soma (cell body) summation, the "storage capacity" of a single neuron increases exponentially. This allows a single DER neuron to outperform a large group of standard perceptrons.
  • Hardware Robustness: Because the system relies on the combination of many binary connections rather than the precise value of a few analog ones, it is inherently more robust to the manufacturing variations (mismatch) common in sub-threshold CMOS circuits.

Key Results

  • Superior Accuracy: Compared to the state-of-the-art $p$-delta algorithm using the same number of synapses, the LSM-DER achieved 3.3x less error in spike train classification and 2.4x less error in the rate approximation task.
  • Resource Efficiency: Even when a group of 60 parallel perceptrons used 60 times more synapses, they still could not match the performance of the proposed dendritically enhanced readout.
  • Silicon Area Savings: In a simulated VLSI implementation (using 0.35 $\mu$m CMOS technology), the DER readout was approximately 4.67 times smaller in area than a comparable PPR readout.
  • Robustness: Monte Carlo simulations showed that when hardware non-idealities were introduced, the error for LSM-DER increased by only ~0.02, whereas the standard PPR error increased by ~0.047, demonstrating better stability.

Limitations and Future Work

The authors acknowledge that the current Network Rewiring (NRW) algorithm can sometimes get trapped in local minima because the search space for optimal wiring is so vast. They suggest that more sophisticated optimization strategies could further improve training.

Additionally, while the paper provides a clear path for VLSI implementation, the learning (the rewiring process) was performed in software. A fully autonomous "on-chip" learning system would require implementing the correlation-based performance index calculation in hardware, which the authors identify as a future goal.

Summary updated

Abstract

In this paper, we describe a new neuro-inspired, hardware-friendly readout stage for the liquid state machine (LSM), a popular model for reservoir computing. Compared to the parallel perceptron architecture trained by the p-delta algorithm, which is the state of the art in terms of performance of readout stages, our readout architecture and learning algorithm can attain better performance with significantly less synaptic resources making it attractive for VLSI implementation. Inspired by the nonlinear properties of dendrites in biological neurons, our readout stage incorporates neurons having multiple dendrites with a lumped nonlinearity (two compartment model). The number of synaptic connections on each branch is significantly lower than the total number of connections from the liquid neurons and the learning algorithm tries to find the best `combination' of input connections on each branch to reduce the error. Hence, the learning involves network rewiring (NRW) of the readout network similar to structural plasticity observed in its biological counterparts. We show that compared to a single perceptron using analog weights, this architecture for the readout can attain, even by using the same number of binary valued synapses, up to 3.3 times less error for a two-class spike train classification problem and 2.4 times less error for an input rate approximation task. Even with 60 times larger synapses, a group of 60 parallel perceptrons cannot attain the performance of the proposed dendritically enhanced readout. An additional advantage of this method for hardware implementations is that the `choice' of connectivity can be easily implemented exploiting address event representation (AER) protocols commonly used in current neuromorphic systems where the connection matrix is stored in memory. Also, due to the use of binary synapses, our proposed method is more robust against statistical variations.
BibTeX
@article{roy2014liquid,
  author = {Subhrajit Roy and Amitava Banerjee and Arindam Basu},
  title = {{Liquid State Machine With Dendritically Enhanced Readout for Low-Power, Neuromorphic VLSI Implementations}},
  journal = {IEEE Trans. Biomed. Circuits Syst.},
  year = {2014},
  url = {https://openreview.net/forum?id=uSOaWhKXyz},
}

Loading related content...

Figure