Federated Neuromorphic Learning of Spiking Neural Networks for Low-Power Edge Intelligence

Nicolas Skatchkovsky, Hyeryung Jang, Osvaldo SimeoneICASSP 2020 · DOI

TL;DR

This paper introduces FL-SNN, a novel federated learning approach for Spiking Neural Networks (SNNs) that enables on-device, low-power, collaborative learning. It addresses the challenge of training SNNs on resource-constrained edge devices by leveraging local feedback and global communication, offering a flexible trade-off between communication load and accuracy.

What Problem Does This Paper Address?

Modern machine learning models, especially deep neural networks, demand significant computational resources, making them unsuitable for low-power edge devices. Spiking Neural Networks (SNNs) offer a promising alternative due to their biological inspiration and energy efficiency, but training them effectively on individual devices with limited data is challenging. Federated Learning (FL) provides a framework for collaborative training without sharing raw data, but its application to SNNs, particularly for online, on-device learning, has been underexplored. This paper bridges this gap by proposing a federated learning scheme specifically designed for SNNs, enabling them to learn collaboratively and efficiently on edge devices.

Methods

The core contribution is the FL-SNN algorithm, which adapts federated learning principles to the unique characteristics of SNNs. Instead of relying on traditional backpropagation, FL-SNN utilizes local feedback signals within each SNN and global feedback through a base station.

Key aspects of the method include:

  • Probabilistic SNN Model: The paper uses a probabilistic Generalized Linear Model (GLM) for SNN neurons, where the spiking probability is a sigmoid function of the neuron's membrane potential.
  • Local Learning Rule: Each device trains its local SNN using an online gradient descent approach based on a log-loss function derived from the SNN's output probabilities. This local learning leverages synaptic traces and feedback mechanisms.
  • Federated Averaging: Periodically, devices communicate their updated model parameters (synaptic weights) to a central base station. The base station then computes a global average of these parameters, which is broadcast back to the devices. This global update guides the collaborative learning process.
  • Flexible Communication: A novel aspect is the ability to trade off communication load and accuracy by selectively exchanging only a subset of synaptic weights, specifically those with the largest gradients. This allows for more frequent updates with reduced bandwidth.

Datasets

The paper uses the MNIST-DVS dataset for evaluating the proposed FL-SNN algorithm. This dataset consists of event-based data from a Dynamic Vision Sensor (DVS) camera, capturing handwritten digits from the MNIST dataset. The data is preprocessed into time-sequences suitable for SNN input.

Key Insights and Ideas

  • SNNs are well-suited for Federated Learning: The paper demonstrates that the principles of FL can be effectively applied to SNNs, enabling collaborative learning on distributed, resource-constrained devices.
  • Local feedback is sufficient for effective SNN training: FL-SNN shows that relying on local learning signals within the SNN, combined with periodic global averaging, can achieve significant learning gains without the need for end-to-end backpropagation across devices.
  • Selective weight exchange offers a practical trade-off: The ability to transmit only a subset of weights (those with the largest gradients) provides a crucial mechanism for managing communication load in federated learning scenarios, which is a significant bottleneck in many FL applications.
  • Frequent, smaller updates are better than infrequent, large ones: The results suggest that transmitting a subset of weights more frequently (smaller communication period $\tau$) leads to better final test accuracy compared to less frequent, but larger, weight exchanges.

Key Results

  • The proposed FL-SNN algorithm demonstrates significant advantages over separate training of SNNs on individual devices.
  • Figure 3 shows that a larger communication period ($\tau$) impairs learning capabilities, leading to higher final test loss. For instance, with $\tau=400$, devices struggle to improve performance without communication.
  • Figure 4 illustrates the trade-off between communication period ($\tau$) and final test accuracy for different communication rates ($r$). It shows that transmitting a subset of weights more frequently enhances cooperative training. For example, with a fixed communication rate of $r=1/8$, increasing the frequency of updates (decreasing $\tau$) generally leads to higher accuracy.

Limitations and Future Work

  • Communication Load: While the paper addresses communication load by selective weight exchange, it acknowledges that transmitting even a subset of weights can still be substantial. Future work could explore more aggressive compression techniques or alternative communication strategies.
  • Hyperparameter Sensitivity: The performance of FL-SNN might be sensitive to hyperparameters like the learning rate ($\alpha$), the eligibility trace decay rate ($\kappa$), and the communication period ($\tau$). Further investigation into robust hyperparameter tuning would be beneficial.
  • Scalability: The experiments were conducted with a small number of devices ($N=2$). Evaluating the scalability of FL-SNN to a larger number of devices is a natural next step.
  • Heterogeneity: While the paper mentions heterogeneous and unbalanced local data sets, further exploration into how FL-SNN performs under extreme data heterogeneity and device capabilities would be valuable.

Summary updated

BibTeX
@inproceedings{skatchkovsky2020federated,
  author = {Nicolas Skatchkovsky and Hyeryung Jang and Osvaldo Simeone},
  title = {{Federated Neuromorphic Learning of Spiking Neural Networks for Low-Power Edge Intelligence}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  doi = {10.1109/icassp40776.2020.9053861},
}
Figure