A surrogate gradient spiking baseline for speech command recognition

Alexandre Bittar, Philip N. GarnerFrontiers in Neuroscience 2022 · DOI

TL;DR

This paper introduces a robust training framework for Spiking Neural Networks (SNNs) using surrogate gradients, enabling them to achieve competitive performance with traditional Artificial Neural Networks (ANNs) on speech command recognition tasks. This work demonstrates that SNNs, with their inherent biological plausibility and potential for energy efficiency, are a viable and promising alternative for complex AI applications like speech processing.

What Problem Does This Paper Address?

Artificial Neural Networks (ANNs) have revolutionized AI, but they rely on continuous-valued neuron activations, which differ significantly from the discrete, event-based "spikes" of biological neurons. Spiking Neural Networks (SNNs) aim to mimic this biological process, offering potential advantages in representational capacity and energy efficiency, especially for time-series data like speech. However, training SNNs has been a major hurdle due to the non-differentiable nature of spike generation. This paper addresses this challenge by developing and evaluating a training methodology that makes SNNs compatible with modern deep learning frameworks, aiming to bridge the gap between SNN research and practical AI applications.

Methods

The core of this paper lies in its approach to training SNNs. The authors leverage surrogate gradient learning, a technique that approximates the non-differentiable spike function with a differentiable surrogate during the backward pass of training. This allows them to use standard gradient descent algorithms like Back-Propagation Through Time (BPTT), making SNNs trainable within existing deep learning frameworks (specifically PyTorch).

Key methodological components include:

  • Physiologically Plausible Neuron Models: They explore and utilize the Leaky Integrate-and-Fire (LIF) and Adaptive Leaky Integrate-and-Fire (adLIF) neuron models. The adLIF model, incorporating subthreshold and spike-triggered adaptation, proved crucial for improved performance.
  • Surrogate Gradient Function: A "boxcar" function is chosen as the surrogate gradient, offering a balance between accuracy and computational efficiency.
  • Discrete Time Formulation: The continuous-time differential equations governing neuron dynamics are converted into discrete-time updates for efficient computation.
  • Readout Layer: A non-spiking readout layer using a cumulative sum of membrane potentials over time was found to yield the best results for classification tasks.
  • ANN Baselines: To provide a fair comparison, they implement and evaluate several standard ANN architectures, including MLPs, RNNs, liBRUs, and GRUs.

Datasets

The study utilizes both spiking and non-spiking versions of established speech datasets:

  • Spiking Heidelberg Digits (SHD): A dataset of spoken digits (0-9) converted into spike trains using the LAUSCHER cochlear model.
  • Spiking Speech Commands (SSC): A dataset derived from the Google Speech Commands v0.2, also converted into spike trains using LAUSCHER.
  • Heidelberg Digits (HD): The original, non-spiking version of the SHD dataset, using filterbank features extracted from audio waveforms.
  • Speech Commands (SC): The original, non-spiking version of the SSC dataset, also using filterbank features.

The use of spiking versions of these datasets is a significant contribution, providing benchmarks specifically designed for evaluating SNNs.

Key Insights and Ideas

  • SNNs Can Surpass ANNs: Contrary to common perception, the paper demonstrates that well-trained SNNs can achieve superior performance compared to comparable ANN architectures on speech recognition tasks. This highlights the potential of SNNs for complex temporal processing.
  • Adaptation is Key: The inclusion of adaptation mechanisms (like in the adLIF model) significantly boosts SNN performance, suggesting that mimicking biological neuron dynamics is crucial for achieving high accuracy.
  • Heterogeneity Matters: The paper suggests that the inherent heterogeneity of spiking neurons (due to trainable parameters) offers an advantage over the homogeneity of standard ANNs, allowing for richer temporal representations with potentially fewer neurons.
  • Surrogate Gradients are Effective: The surrogate gradient approach proves to be a viable and powerful method for training SNNs, making them accessible within mainstream deep learning frameworks.
  • Energy Efficiency Potential: The study strongly implies that SNNs, due to their sparse and event-driven nature, offer substantial energy savings compared to ANNs, making them ideal for low-power devices.

Key Results

  • Competitive Performance: SNNs achieved state-of-the-art or near state-of-the-art results on the SHD and SSC datasets, often outperforming comparable ANN baselines. For instance, on the SHD dataset, their best SNN achieved 94.62% accuracy, surpassing the ANN SOTA of 92.4%.
  • Lighter Architectures: Non-recurrent SNNs with adaptation (adLIF) were shown to outperform larger, recurrent ANNs, demonstrating the efficiency gains from biologically inspired models.
  • Robustness to Sparsity: Experiments showed that SNNs could maintain high accuracy even with significant reductions in network connectivity, indicating efficient parameter utilization.
  • Adaptation vs. Recurrence: On some tasks, adding adaptation to SNNs provided performance gains comparable to adding recurrent connections, but with fewer trainable parameters.

Limitations and Future Work

  • Neuron Model Complexity: While more complex, nonlinear neuron models (like Izhikevich and AdEx) were explored, they did not consistently outperform the simpler adLIF model. The authors suggest this might be due to compatibility issues with gradient descent or the nature of the spiking datasets.
  • Dataset Conversion: The conversion of audio waveforms to spike trains using LAUSCHER might lead to some information loss compared to traditional filterbank features, as observed in the HD dataset results.
  • Biological Plausibility vs. Performance: The study prioritizes compatibility with deep learning frameworks, which may not fully capture all aspects of biological learning rules.
  • Future Work: The authors suggest further research into more complex neuron models, exploring bio-inspired learning rules beyond surrogate gradients, and applying these SNNs to more advanced speech processing tasks like Automatic Speech Recognition (ASR). They also highlight the potential for SNNs to aid in understanding biological mechanisms.

Summary updated

Abstract

Artificial neural networks (ANNs) are the basis of recent advances in artificial intelligence (AI); they typically use real valued neuron responses. By contrast, biological neurons are known to operate using spike trains. In principle, spiking neural networks (SNNs) may have a greater representational capability than ANNs, especially for time series such as speech; however their adoption has been held back by both a lack of stable training algorithms and a lack of compatible baselines. We begin with a fairly thorough review of literature around the conjunction of ANNs and SNNs. Focusing on surrogate gradient approaches, we proceed to define a simple but relevant evaluation based on recent speech command tasks. After evaluating a representative selection of architectures, we show that a combination of adaptation, recurrence and surrogate gradients can yield light spiking architectures that are not only able to compete with ANN solutions, but also retain a high degree of compatibility with them in modern deep learning frameworks. We conclude tangibly that SNNs are appropriate for future research in AI, in particular for speech processing applications, and more speculatively that they may also assist in inference about biological function.
BibTeX
@inproceedings{bittar2022surrogate,
  author = {Alexandre Bittar and Philip N. Garner},
  title = {{A surrogate gradient spiking baseline for speech command recognition}},
  booktitle = {Frontiers in Neuroscience},
  year = {2022},
  doi = {10.3389/fnins.2022.865897},
}
Figure