When Spiking Neural Networks Meet Temporal Attention Image Decoding and Adaptive Spiking Neuron

Xuerui Qiu, Zheng Luan, Zhaorui Wang, Rui-Jie ZhuSubmission to Tiny Papers @ ICLR

Introduction

Spiking Neural Networks (SNNs) offer a biologically plausible, energy-efficient alternative to traditional artificial neural networks by communicating via discrete "spikes" over time. However, unlocking their full potential for complex vision tasks has been challenging. This paper introduces two key innovations: ALIF, a neuron model with learnable thresholds and time constants, and TAID, a decoding mechanism that uses temporal attention to reconstruct high-quality images from spikes. Together, these methods push SNNs to state-of-the-art performance in both image classification and generative modeling.

The Problem with Current SNNs

SNNs face two major hurdles in computer vision:

  1. Rigid Neuron Dynamics: Most SNNs use the Leaky-Integrate-and-Fire (LIF) model with fixed parameters. They often ignore "threshold dynamics"—the biological principle that a neuron's firing threshold should adapt based on its activity.
  2. Temporal Neglect in Decoding: When using SNNs to generate images (such as in a Variational Autoencoder), existing methods often fail to fully exploit the rich temporal information contained in the spike sequences. This typically leads to blurry or lower-quality image reconstructions.

Proposed Methods: ALIF and TAID

To address these issues, the authors propose a dual-pronged solution targeting both the neuron level and the decoding architecture.

1. Adaptive Leaky-Integrate-and-Fire (ALIF)

In a standard LIF neuron, the membrane potential decays at a fixed rate ($\tau$) and fires a spike once it hits a fixed threshold ($V_{th}$). The authors make both $ au$ and $V_{th}$ learnable parameters. By deriving the backpropagation chain rules for these parameters using surrogate gradients (to handle the non-differentiable nature of spikes), the network can "tune" its own firing sensitivity and memory duration during training.

2. Temporal Attention Image Decoding (TAID)

To turn a sequence of spikes back into a static image, the authors developed TAID. Instead of simply averaging spikes over time, TAID:

  • Squeezes the output spike tensor to find mean activity across space and channels.
  • Uses a learnable weight matrix to identify correlations between specific time steps and channels.
  • Applies a Sigmoid-based fusion matrix to weight the original spike sequences, effectively "paying attention" to the most informative moments in the temporal sequence to reconstruct the final image.
Comparison of FSVAE and the proposed TAID architecture

Key Results and Insights

The researchers evaluated their methods on standard benchmarks including MNIST, CIFAR-10, and CelebA.

  • Classification: The ALIF model achieved 99.78% accuracy on MNIST and 93.89% on CIFAR-10, outperforming previous state-of-the-art SNN models like PLIF. Notably, these results were achieved using only 8 simulation time steps, showing that high performance doesn't necessarily require long, energy-intensive processing windows.
  • Generation: On the CelebA dataset, the TAID method achieved an Inception Score (IS) of 4.31, significantly higher than the previous FSVAE (3.69) and even a standard ANN VAE (3.23).
Visual comparison of generated faces on CelebA using ANN VAE, FSVAE, and TAID

Core Takeaways

  • Thresholds Matter: Ablation studies show that making the threshold ($V_{th}$) learnable provides a significant boost in accuracy compared to keeping it fixed. This suggests that "neural flexibility" is just as important as weight optimization.
  • Time is a Feature, Not a Burden: While many researchers treat the temporal dimension of SNNs as a computational cost to be minimized, TAID demonstrates that this dimension contains vital structural information that can significantly improve generative tasks.
  • SNNs are Closing the Gap: By outperforming standard Artificial Neural Networks (ANNs) in certain generative metrics, the paper suggests that SNNs are becoming viable alternatives for complex creative AI tasks, not just simple classification.

Limitations and Future Work

While ALIF improves threshold dynamics, it still follows a relatively simple iterative version of the LIF model. The paper does not explore how these learnable thresholds behave in much deeper architectures, such as Spiking Transformers. Furthermore, while TAID excels at static image decoding, its potential for actual temporal data (like video or DVS camera events) remains an open question for future research. Finally, the paper focuses on algorithmic performance; the actual energy savings on neuromorphic hardware (like Loihi or TrueNorth) were not measured.

Summary updated

Abstract

Spiking Neural Networks (SNNs) are capable of encoding and processing temporal information in a biologically plausible way. However, most existing SNN-based methods for image tasks do not fully exploit this feature. Moreover, they often overlook the role of adaptive threshold in spiking neurons, which can enhance their dynamic behavior and learning ability. To address these issues, we propose a novel method for image decoding based on temporal attention (TAID) and an adaptive Leaky-Integrate-and-Fire (ALIF) neuron model. Our method leverages the temporal information of SNN outputs to generate high-quality images that surpass the state-of-the-art (SOTA) in terms of Inception score, Fréchet Inception Distance, and Fréchet Autoencoder Distance. Furthermore, our ALIF neuron model achieves remarkable classification accuracy on MNIST (99.78%) and CIFAR-10 (93.89%) datasets, demonstrating the effectiveness of learning adaptive thresholds for spiking neurons.
BibTeX
@misc{qiu2024when,
  author = {Xuerui Qiu and Zheng Luan and Zhaorui Wang and Rui-Jie Zhu},
  title = {{When Spiking Neural Networks Meet Temporal Attention Image Decoding and Adaptive Spiking Neuron}},
  year = {2024},
  eprint = {2406.03046},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.03046},
}
Figure