A Novel Explainable Out-of-Distribution Detection Approach for Spiking Neural Networks

Aitor Martínez-Seras, Javier Del Ser, Jesus L. Lobo, Pablo García Bringas, Nikola K. KasabovarXiv 2022 · OpenReview

TL;DR

This paper introduces a novel method for detecting Out-of-Distribution (OoD) data in Spiking Neural Networks (SNNs) by analyzing "spike count patterns" within the network's hidden layers. Crucially, it also provides an explainable AI component that generates heatmaps highlighting which input features contribute most to an OoD detection, making SNNs more trustworthy in real-world applications.

What Problem Does This Paper Address?

Spiking Neural Networks (SNNs) offer advantages like energy efficiency and temporal modeling but, like other neural networks, can struggle when presented with data that differs from their training distribution (Out-of-Distribution or OoD data). This is a critical issue for trustworthiness, especially in applications like autonomous driving or medical diagnosis where misclassifications can have serious consequences. Existing OoD detection methods often don't directly apply to SNNs, and even when they do, they typically lack explainability, leaving users in the dark about why a sample is flagged as OoD. This paper aims to fill this gap by developing an SNN-specific, explainable OoD detection method.

Methods

The core of the proposed method, dubbed the Spike Count Pattern (SCP) based detector, relies on analyzing the patterns of neuron firings (spike counts) in the SNN's hidden layers.

  1. Training Stage:
    • The SNN is trained on the in-distribution (ID) dataset.
  2. For each class, a subset of training examples is used to extract spike counts from the layer before the final classification layer.
  3. These spike counts are then clustered to identify "archetypes" or representative patterns for each class. These archetypes are essentially the centroids of these clusters.
  4. Detection Stage:
    • When a new test sample arrives, it's passed through the trained SNN to get a predicted class.
  5. The spike count pattern of this test sample is computed.
  6. An OoD score is calculated based on the distance (specifically, L1 or Manhattan distance) between the test sample's spike count pattern and the archetypes of its predicted class.
  7. If this distance exceeds a class-specific threshold, the sample is flagged as OoD.
  8. Explainability (Local Relevance Attribution):
    • To explain why a sample is flagged as OoD, the method backpropagates the spike counts from the detection layer back to the input layer using the network's weights.
  9. This process reconstructs the spike counts in the input space.
  10. An attribution vector (which can be visualized as a heatmap for images) is then generated by calculating the L1 distance between the reconstructed spike counts of the input and the reconstructed spike counts of the closest archetype. This highlights which input features were most influential in the OoD detection.

Datasets

The paper uses standard image classification datasets for training and evaluation:

  • In-distribution datasets: MNIST, Fashion MNIST, KMNIST, EMNIST Letters.
  • Out-of-distribution datasets: notMNIST, Omniglot, CIFAR10-BW, as well as combinations of the in-distribution datasets themselves.
  • For qualitative evaluation of the attribution method, they also use MNIST-Square and MNIST-C, which are modified versions of MNIST with specific artifacts.

Key Insights and Ideas

  • Spike Count Patterns as Signatures: The paper demonstrates that the patterns of neuron firings (spike counts) in SNNs can serve as unique signatures for different classes, even for data that is out-of-distribution.
  • SNNs are Not Immune to OoD: Despite their unique processing, SNNs face the same fundamental challenge as traditional NNs when encountering unseen data distributions.
  • Logits Aren't Always Enough: The study suggests that relying solely on output layer activations (logits) for OoD detection in SNNs might not be sufficient, as demonstrated by the performance of baseline methods.
  • Explainability is Key for Trust: Providing heatmaps that pinpoint the features driving an OoD detection significantly enhances the trustworthiness and interpretability of SNNs.
  • Depth vs. Attribution Quality: A trade-off exists where deeper fully-connected layers in SNNs can improve detection but potentially degrade the clarity of attribution heatmaps.

Key Results

  • Competitive Detection Performance: The proposed SCP-based detector performs competitively, and often superiorly, to established post-hoc OoD detection methods adapted for SNNs across various datasets and architectures (Fully Connected SNN and Convolutional SNN).
  • Statistical Significance: Bayesian analysis confirms that the SCP-based detector often outperforms ODIN (a strong baseline) with statistical significance.
  • Effective Attribution: The relevance attribution method successfully highlights artificial artifacts introduced into images, demonstrating its ability to pinpoint features that push a sample towards being classified as OoD. For instance, in Figure 8, the heatmaps clearly highlight the added square or zigzag patterns on digits.
  • Performance Metrics: The paper reports results using AUROC, AUPR, and FPR95 scores, showing the SCP detector achieving high scores in many scenarios. For example, in Table 2, the SCP detector often achieves the highest AUROC scores.

Limitations and Future Work

  • Network Depth: The quality of attribution heatmaps degrades as the depth of the fully connected part of the SNN increases, leading to noisier visualizations.
  • Feature Extractor Dependence: The effectiveness of the attribution method is tied to the quality of features extracted by earlier layers (e.g., convolutional layers). If these features don't capture the OoD nature, the attribution might be less informative.
  • Intra-cluster Variance: High variance within clusters of spike count patterns can sometimes mask OoD characteristics, as a reconstructed centroid might coincidentally match the input's reconstruction.
  • Future Work: The authors plan to extend the SCP detector to other SNN encoding methods, mitigate the limitations of attribution heatmaps, explore generative SNNs for counterfactual explanations, and apply the method to temporal and spatio-temporal streaming data (e.g., from dynamic vision sensors, EEG, financial data).

Summary updated

Abstract

Research around Spiking Neural Networks has ignited during the last years due to their advantages when compared to traditional neural networks, including their efficient processing and inherent ability to model complex temporal dynamics. Despite these differences, Spiking Neural Networks face similar issues than other neural computation counterparts when deployed in real-world settings. This work addresses one of the practical circumstances that can hinder the trustworthiness of this family of models: the possibility of querying a trained model with samples far from the distribution of its training data (also referred to as Out-of-Distribution or OoD data). Specifically, this work presents a novel OoD detector that can identify whether test examples input to a Spiking Neural Network belong to the distribution of the data over which it was trained. For this purpose, we characterize the internal activations of the hidden layers of the network in the form of spike count patterns, which lay a basis for determining when the activations induced by a test instance is atypical. Furthermore, a local explanation method is devised to produce attribution maps revealing which parts of the input instance push most towards the detection of an example as an OoD sample. Experimental results are performed over several image classification datasets to compare the proposed detector to other OoD detection schemes from the literature. As the obtained results clearly show, the proposed detector performs competitively against such alternative schemes, and produces relevance attribution maps that conform to expectations for synthetically created OoD instances.
BibTeX
@misc{martinezseras2022novel,
  author = {Aitor Martínez-Seras and Javier Del Ser and Jesus L. Lobo and Pablo García Bringas and Nikola K. Kasabov},
  title = {{A Novel Explainable Out-of-Distribution Detection Approach for Spiking Neural Networks}},
  year = {2022},
  eprint = {2210.00894},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2210.00894},
}
Figure