Spiking-Diffusion: Vector Quantized Discrete Diffusion Model with Spiking Neural Networks

Mingxuan Liu, Jie Gan, Rui Wen, Tao Li, Yongli Chen, Hong Chen

TL;DR

This paper introduces "Spiking-Diffusion," the first diffusion model built entirely from Spiking Neural Networks (SNNs). By combining a novel SNN-based Vector Quantized Variational Autoencoder (VQ-SVAE) with a discrete diffusion process, they achieve state-of-the-art image generation results for SNNs, paving the way for more energy-efficient neuromorphic AI.

What Problem Does This Paper Address?

Spiking Neural Networks (SNNs) are highly promising for energy-efficient AI on neuromorphic hardware due to their event-driven, binary nature. While successful in tasks like classification, their application in complex generative tasks, such as image generation, has been limited. Existing SNN-based generative models, like Spiking-GAN and fully spiking VAEs, struggle to match the quality of their Artificial Neural Network (ANN) counterparts, often due to limitations in generative capacity and the difficulty of handling continuous signals with discrete spike events. This paper aims to bridge this gap by developing a diffusion model that leverages the strengths of SNNs for high-quality image generation.

Methods

The proposed "Spiking-Diffusion" model operates in two main stages:

  1. SNN-based Vector Quantized Variational Autoencoder (VQ-SVAE): This component learns a discrete latent representation of images.
  2. Encoding: Images are first converted into spike sequences. To manage memory, these sequences are encoded using a combination of Spike Firing Rate (SFR) and Postsynaptic Potential (PSP). A learnable parameter k balances the contribution of SFR and PSP.
  3. Quantization: The encoded features are then quantized into a discrete latent space using a codebook, similar to standard VQ-VAEs.
  4. Adaptive Spike Generator (ASG): A crucial innovation is the ASG, an SNN layer that learns to reconstruct the original spike sequences from the quantized latent representation. This addresses the information loss that can occur when converting discrete latent codes back into spike trains.
  5. Spiking Diffusion Image Decoder (SDID): This stage models the discrete latent space using a diffusion process.
  6. Forward Process: A Markov chain with an absorbing state is used to gradually add "masks" to the discrete latent representations over a fixed number of timesteps. This process is designed to be compatible with SNNs.
  7. Reverse Process: An SNN-based decoder then learns to reverse this masking process, gradually denoising the latent representation to generate a high-quality image. The model is trained using a loss function that includes reconstruction terms and a term based on the PSP kernel for the ASG.

Datasets

The model was evaluated on the following standard image datasets:

  • MNIST
  • FMNIST (Fashion-MNIST)
  • KMNIST (Kuzushiji-MNIST)
  • Letters
  • Cifar10

Key Insights and Ideas

  • Bridging SNNs and Diffusion Models: This work is the first to successfully integrate the powerful generative capabilities of diffusion models with the energy efficiency of SNNs, creating a fully SNN-based diffusion pipeline.
  • VQ-SVAE for Discrete Latent Spaces: Developing a VQ-VAE entirely within SNNs is challenging. The paper's approach of combining SFR and PSP for feature encoding and using an ASG to recover spike trains is a novel solution.
  • Absorbing State Diffusion for SNNs: Adapting diffusion models to discrete SNNs requires careful design. The use of an absorbing state in the forward diffusion process is a key element that makes the discrete diffusion compatible with SNN architectures.
  • Improved Generative Quality: The proposed method demonstrates that SNNs can achieve competitive image generation quality, surpassing previous SNN-based generative models.

Key Results

  • State-of-the-Art SNN Generation: Spiking-Diffusion outperforms the existing SNN-based generative model, FSVAE, across multiple datasets.
  • Significant FID Improvements: The paper reports substantial reductions in Fréchet Inception Distance (FID) compared to the state-of-the-art:
  • MNIST: 37.50 (58.60% reduction)
    • FMNIST: 91.98 (18.75% reduction)
    • KMNIST: 59.23 (64.51% reduction)
    • Letters: 67.41 (29.75% reduction)
    • Cifar10: 120.5 (44.88% reduction)
  • Qualitative Improvements: Generated images from Spiking-Diffusion are described as clearer with more obvious boundaries compared to the blurrier images produced by FSVAE.

Limitations and Future Work

  • Limitations: While the paper demonstrates strong performance, the authors acknowledge that the quality of generated images still falls short of state-of-the-art ANN-based diffusion models. The complexity of training and the inherent limitations of SNNs for capturing very fine details might contribute to this.
  • Future Work: The authors suggest that future research should focus on scaling up SNN generative models to handle larger and more complex datasets, potentially exploring more advanced SNN architectures or training techniques.

Summary updated

Abstract

Spiking neural networks (SNNs) have tremendous potential for energy-efficient neuromorphic chips due to their binary and event-driven architecture. SNNs have been primarily used in classification tasks, but limited exploration on image generation tasks. To fill the gap, we propose a Spiking-Diffusion model, which is based on the vector quantized discrete diffusion model. First, we develop a vector quantized variational autoencoder with SNNs (VQ-SVAE) to learn a discrete latent space for images. In VQ-SVAE, image features are encoded using both the spike firing rate and postsynaptic potential, and an adaptive spike generator is designed to restore embedding features in the form of spike trains. Next, we perform absorbing state diffusion in the discrete latent space and construct a spiking diffusion image decoder (SDID) with SNNs to denoise the image. Our work is the first to build the diffusion model entirely from SNN layers. Experimental results on MNIST, FMNIST, KMNIST, Letters, and Cifar10 demonstrate that Spiking-Diffusion outperforms the existing SNN-based generation model. We achieve FIDs of 37.50, 91.98, 59.23, 67.41, and 120.5 on the above datasets respectively, with reductions of 58.60%, 18.75%, 64.51%, 29.75%, and 44.88% in FIDs compared with the state-of-art work. Our code will be available at https://github.com/Arktis2022/Spiking-Diffusion.
BibTeX
@misc{liu2023spiking,
  author = {Mingxuan Liu and Jie Gan and Rui Wen and Tao Li and Yongli Chen and Hong Chen},
  title = {{Spiking-Diffusion: Vector Quantized Discrete Diffusion Model with Spiking Neural Networks}},
  year = {2023},
  eprint = {2308.10187},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2308.10187},
}
Figure