Online Processing of Multiple Inputs in a Sparsely-Connected Recurrent Neural Network

Julien Mayor, Wulfram GerstnerICANN 2003 · DOI

TL;DR

This paper demonstrates that sparsely-connected spiking neural networks, when tuned to an asynchronous irregular firing regime, can effectively process multiple superimposed input streams simultaneously. The network's inherent dynamics provide a rich, high-dimensional representation of the input, allowing simple linear readouts to extract individual signals with remarkable accuracy, even when information has diffused throughout the network.

What Problem Does This Paper Address?

The paper tackles the challenge of how recurrent neural networks (RNNs), particularly those inspired by biological neural systems (like Liquid State Machines or Echo State Networks), can handle and process multiple, distinct streams of information concurrently. A key difficulty is that in such networks, information about individual inputs can become distributed and intertwined across many neurons, making it hard to isolate and retrieve specific signals. The authors aim to show that despite this diffusion, effective information extraction is possible, and that the network's internal dynamics can perform the complex transformations often achieved by more sophisticated kernel methods.

Methods

The authors employ a sparsely-connected network of 200 leaky integrate-and-fire (IF) neurons, with 80% excitatory and 20% inhibitory. This network is specifically tuned to operate in an asynchronous irregular firing regime, which is crucial for generating rich, non-linear dynamics. They inject multiple independent input signals, each targeting a disjoint group of neurons. To read out the information, they use simple linear "readout structures" that are trained via linear regression to predict past input signals (i.e., Input(t - T)). The error minimized is the squared difference between the readout's output and the target input. This approach allows them to assess how well the network preserves and makes accessible information about past inputs.

Datasets

Not discussed in the paper. The paper focuses on synthetic input signals generated from bounded random walks, rather than pre-existing datasets.

Key Insights and Ideas

  • Rich Dynamics as a Feature Extractor: The core idea is that the asynchronous irregular firing regime of the spiking neural network itself acts as a powerful, non-linear feature extractor. This dynamic reservoir implicitly performs the "dimensional blow-up" that is often a prerequisite for successful signal processing in kernel methods.
  • Information Diffusion is Manageable: Even though information about individual inputs spreads throughout the network, simple linear readouts are capable of disentangling and retrieving these signals. This suggests that the network's connectivity and dynamics create a structured representation that is amenable to linear decoding.
  • Linear Readouts are Sufficiently Powerful: The paper demonstrates that simple linear regression readouts can achieve performance comparable to complex kernel-based methods (like Support Vector Machines with polynomial kernels) for this task. This highlights the effectiveness of the network's internal representation.
  • Trade-off in Information Retrieval: When trying to retrieve past information, the network and readout learn to balance extracting degraded past information with reading the present input, which is correlated with the past. This optimal trade-off is influenced by the statistics of the target signals.

Key Results

  • The network can successfully process multiple superimposed input streams simultaneously, even with eight independent signals.
  • Information remains extractable for delays up to approximately 150ms, demonstrating effective short-term memory.
  • Analysis of readout weights shows that information is not strictly localized but rather diffused across the network, yet still accessible.
  • Linear readouts perform comparably to, and in some cases better than, kernel-based methods like SVMs, suggesting the network's internal dynamics provide the necessary high-dimensional representation.

Limitations and Future Work

  • Biological Plausibility of Readouts: The paper acknowledges that the "optimal" linear readouts, which access sub-threshold membrane potentials, are not biologically plausible. Future work could explore readouts based solely on spike times.
  • Overfitting with Complex Kernels: The study observed that more complex kernel methods (e.g., higher-degree polynomial kernels) showed signs of over-fitting, reinforcing the idea that simple linear readouts are effective when paired with the network's rich dynamics.
  • Influence of Target Statistics: The performance is noted to be dependent on the underlying statistics of the target signals, suggesting that further investigation into how target properties influence retrieval is warranted.

Summary updated

BibTeX
@inproceedings{mayor2003online,
  author = {Julien Mayor and Wulfram Gerstner},
  title = {{Online Processing of Multiple Inputs in a Sparsely-Connected Recurrent Neural Network}},
  booktitle = {ICANN},
  year = {2003},
  doi = {10.1007/3-540-44989-2\_100},
}
Figure