DTA: Dual Temporal-channel-wise Attention for Spiking Neural Networks

Minje Kim, Minjun Kim, Xu Yang

TL;DR

Spiking Neural Networks (SNNs) offer a low-power alternative to traditional AI, but they often struggle to effectively process information over time. This paper introduces Dual Temporal-channel-wise Attention (DTA), a novel mechanism that combines "identical" and "non-identical" attention operations to capture both correlations and complex dependencies across time and channels. The result is a state-of-the-art SNN that outperforms previous models on both static images and dynamic, event-based datasets with fewer processing steps.

What Problem Does This Paper Address?

While standard Artificial Neural Networks (ANNs) have revolutionized computer vision, they are power-hungry. SNNs are biologically inspired "event-driven" networks that communicate via discrete spikes, making them incredibly energy-efficient on specialized neuromorphic hardware.

However, SNNs face a "temporal bottleneck." To perform well, they must selectively attend to important information across both time (when a spike occurs) and channels (which neuron is spiking). Existing attention mechanisms for SNNs typically choose one of two paths:

  1. Identical operations: Applying the same mathematical operation across dimensions to find correlations.
  2. Non-identical operations: Applying different operations to interpret complex, non-linear dependencies.

The authors argue that choosing just one limits the network's expressive power. They set out to create a unified block that leverages the strengths of both perspectives to create a richer "spike representation."

Methods

The authors propose the Dual Temporal-channel-wise Attention (DTA) mechanism, which consists of two independent but complementary modules:

  • T-XA (Identical Cross Attention): This module focuses on the correlation between temporal and channel information. It uses "identical" operations—Temporal Local Attention (TLA) and Channel Local Attention (CLA)—to ensure fine-grained alignment between when and where spikes are occurring.
  • T-NA (Non-identical Attention): This module focuses on dependency. It uses "non-identical" operations to capture both local and global relationships. It employs a mix of depth-wise, dilated, and point-wise convolutions (Local Attention) alongside an MLP bottleneck (Global Attention) to handle the combined temporal-channel dimensions.

These two modules are integrated using element-wise multiplication and a sigmoid function. The entire DTA block is designed to be "plug-and-play," embedded here into a modified ResNet architecture (MS-ResNet). The network is trained directly using Surrogate Gradients (SG), a technique that bypasses the non-differentiable nature of discrete spikes.

Datasets

The researchers evaluated DTA on four major benchmarks:

  • Static Datasets: CIFAR-10, CIFAR-100, and the large-scale ImageNet-1k. These test the model's ability to classify standard images.
  • Dynamic Dataset: CIFAR10-DVS. This is a neuromorphic dataset where images are converted into a stream of events (spikes) using a Dynamic Vision Sensor, testing the model's ability to handle "true" temporal data.

Key Insights and Ideas

  • The Power of "Dual" Perspectives: The core "aha moment" is that correlation (identical ops) and dependency (non-identical ops) are not redundant; they provide distinct information. Combining them allows a single DTA block to outperform previous models that used multiple, more complex attention layers.
  • Efficiency in Time: SNNs usually require many "time steps" to reach high accuracy, which increases latency. DTA achieves top-tier results in as few as 4 to 6 time steps, making it highly practical for real-time applications.
  • Better Feature Localization: Using Grad-CAM visualizations, the authors showed that DTA is significantly better at "focusing" on the actual object in an image (like a bird or a car) compared to models using only one type of attention, especially in noisy dynamic data.

Key Results

DTA-SNN achieved State-of-the-Art (SOTA) performance across the board:

  • CIFAR-10: 96.73% accuracy (at 6 time steps), surpassing the previous best (GAC-SNN).
  • CIFAR-100: 81.16% accuracy, a significant jump over prior SNN methods.
  • ImageNet-1k: 71.29% top-1 accuracy, proving the architecture scales to complex, real-world data.
  • CIFAR10-DVS: 81.3% accuracy, outperforming even recent transformer-based SNN architectures.

Crucially, the model achieved these results with only a marginal increase in parameters (approx. 0.09M more than the baseline), maintaining the energy-efficiency advantage of SNNs.

Limitations and Future Work

While the paper demonstrates impressive results, a few areas remain for exploration:

  • Hardware Implementation: While SNNs are theoretically efficient, the paper evaluates them on standard GPUs (NVIDIA A100). The true energy savings would be best demonstrated on neuromorphic chips like Intel’s Loihi or IBM’s TrueNorth.
  • Complexity vs. Latency: Although the model uses fewer time steps, the internal operations of the T-NA module (with multiple convolution types) add some computational complexity per step.
  • Task Generalization: The study focuses on classification. Future work could investigate how DTA performs in more complex spatio-temporal tasks like object detection or action recognition in video streams.

Summary updated

Abstract

Spiking Neural Networks (SNNs) present a more energy-efficient alternative to Artificial Neural Networks (ANNs) by harnessing spatio-temporal dynamics and event-driven spikes. Effective utilization of temporal information is crucial for SNNs, leading to the exploration of attention mechanisms to enhance this capability. Conventional attention operations either apply identical operation or employ non-identical operations across target dimensions. We identify that these approaches provide distinct perspectives on temporal information. To leverage the strengths of both operations, we propose a novel Dual Temporal-channel-wise Attention (DTA) mechanism that integrates both identical/non-identical attention strategies. To the best of our knowledge, this is the first attempt to concentrate on both the correlation and dependency of temporal-channel using both identical and non-identical attention operations. Experimental results demonstrate that the DTA mechanism achieves state-of-the-art performance on both static datasets (CIFAR10, CIFAR100, ImageNet-1k) and dynamic dataset (CIFAR10-DVS), elevating spike representation and capturing complex temporal-channel relationship. We open-source our code: https://github.com/MnJnKIM/DTA-SNN.
BibTeX
@misc{kim2025dta,
  author = {Minje Kim and Minjun Kim and Xu Yang},
  title = {{DTA: Dual Temporal-channel-wise Attention for Spiking Neural Networks}},
  year = {2025},
  eprint = {2503.10052},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.10052},
  howpublished = {arXiv},
}
Figure