SNN-RAT: Robustness-enhanced Spiking Neural Network through Regularized Adversarial Training

Jianhao Ding, Tong Bu, Zhaofei Yu, Tiejun Huang, Jian K. LiuNeurIPS 2022 · OpenReview

Spiking Neural Networks (SNNs) are frequently praised for their biological plausibility and exceptional energy efficiency. Because they process information using discrete "spikes" over time, many researchers historically believed SNNs were inherently resistant to adversarial attacks—the tiny, intentional perturbations designed to trick AI systems.

However, recent research reveals this perceived robustness is largely an illusion. This paper introduces SNN-RAT, a novel training framework that combines a theoretical "spiking Lipschitz constant" with mixed adversarial training techniques to significantly harden SNNs against sophisticated gradient-based attacks.

The Illusion of SNN Robustness

The difficulty attackers face when targeting SNNs is not due to the network's inherent robustness, but rather obfuscated gradients. Because discrete spikes (0s and 1s) are non-differentiable, attackers cannot easily calculate a gradient to find the optimal perturbation.

Once attackers utilize differentiable approximations—such as surrogate gradients—SNNs become highly vulnerable. Furthermore, while traditional Artificial Neural Networks (ANNs) rely on well-established theoretical frameworks for robustness (like Lipschitz analysis), SNNs have lacked a similar mathematical foundation to understand how noise propagates through their unique temporal layers.

The SNN-RAT Methodology

To address these vulnerabilities, the authors propose a two-pronged approach called Regularized Adversarial Training (RAT):

1. Theoretical Foundation: The Spiking Lipschitz Constant

The authors derive a Lipschitz constant specifically tailored for spike representations. In deep learning, a Lipschitz constant bounds how much a layer's output can change relative to its input. By mathematically defining this for SNNs, the researchers demonstrate that robustness can be improved by constraining the "spectral norm" (the largest singular value) of the network's weight matrices.

2. The RAT Framework

Building on this theory, the RAT framework employs two primary mechanisms:

  • REG (Regularization): To keep the spiking Lipschitz constant in check, the framework applies orthogonal regularization. This forces the network's weights to remain in a state where they do not excessively amplify input perturbations as the signal passes through successive layers.
  • MIX (Mixed Adversarial Training): Rather than training against a single type of attack, the model is trained on a "cocktail" of adversarial examples. Perturbations are generated using two different gradient approximations: BPTT (Backpropagation Through Time) and BPTR (Backpropagation Through Rate). Training on this mixture helps the model generalize across different "adversarial neighborhoods."

Experimental Setup and Key Insights

The researchers evaluated SNN-RAT on standard computer vision benchmarks (CIFAR-10 and CIFAR-100) using common SNN architectures like VGG-11 and WideResNet-16 (WRN-16).

During their evaluation, several key insights emerged:

  • Time Matters: Attacks utilizing temporal information (BPTT) are generally more lethal than those relying solely on average firing rates (BPTR).
  • Orthogonality is Key: Forcing weight matrices toward orthogonality successfully prevents the "explosion" of perturbations, ensuring small input changes don't drastically alter the final spike train.
  • Efficiency via "MIX": Training on a mixture of computationally cheap single-step attacks provides surprisingly strong defense against much harder multi-step attacks (like Projected Gradient Descent, or PGD).

Key Results

The implementation of SNN-RAT yields state-of-the-art robustness for Spiking Neural Networks, significantly outperforming previous defense methods with minimal trade-offs.

Robustness comparison of VANILLA vs RAT models across white-box and black-box attacks.
  • Massive Robustness Gains: On CIFAR-10 using VGG-11, a vanilla SNN's accuracy plummets to a mere 0.04% under a strong PGD attack. Under the exact same attack, the SNN-RAT model maintains 21.16% accuracy—a massive leap in adversarial defense.
  • Black-Box Immunity: SNN-RAT proves incredibly effective against "black-box" attacks (where the attacker lacks knowledge of the model's internal weights). While a vanilla model might fall to 4% accuracy in these scenarios, the RAT-trained model consistently stays above 75%.
  • Minimal "Clean" Accuracy Trade-off: Enhancing robustness often degrades a model's performance on normal, unperturbed data. SNN-RAT keeps this "accuracy tax" remarkably low, dropping only from ~93% to ~90% on clean CIFAR-10 images.

Limitations and Future Directions

While SNN-RAT marks a significant step forward, challenges remain. Adversarial training inherently introduces computational overhead; even with efficient single-step attacks, generating adversarial examples for every batch slows down training.

Additionally, the authors note a Batch Normalization (BN) dilemma: while BN is excellent for training speed, it may actively harm SNN robustness. Future research will need to explore how to train high-performance, robust SNNs without relying on BN, as well as scaling these defenses to more complex datasets like ImageNet or event-based neuromorphic datasets (e.g., DVS-Gesture).

Summary updated

Abstract

Spiking neural networks (SNNs) are promising to be widely deployed in real-time and safety-critical applications with the advance of neuromorphic computing. Recent work has demonstrated the insensitivity of SNNs to small random perturbations due to the discrete internal information representation. The variety of training algorithms and the involvement of the temporal dimension pose more threats to the robustness of SNNs than that of typical neural networks. We account for the vulnerability of SNNs by constructing adversaries based on different differentiable approximation techniques. By deriving a Lipschitz constant specifically for the spike representation, we first theoretically answer the question of how much adversarial invulnerability is retained in SNNs. Hence, to defend against the broad attack methods, we propose a regularized adversarial training scheme with low computational overheads. SNNs can benefit from the constraint of the perturbed spike distance's amplification and the generalization on multiple adversarial $\epsilon$-neighbourhoods. Our experiments on the image recognition benchmarks have proven that our training scheme can defend against powerful adversarial attacks crafted from strong differentiable approximations. To be specific, our approach makes the black-box attacks of the Projected Gradient Descent attack nearly ineffective. We believe that our work will facilitate the spread of SNNs for safety-critical applications and help understand the robustness of the human brain.
BibTeX
@inproceedings{ding2022snn,
  author = {Jianhao Ding and Tong Bu and Zhaofei Yu and Tiejun Huang and Jian K. Liu},
  title = {{SNN-RAT: Robustness-enhanced Spiking Neural Network through Regularized Adversarial Training}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=Sy1lwewvN7},
}
Figure