Spiking Neural Networks with Temporal Attention-Guided Adaptive Fusion for imbalanced Multi-modal Learning

Jiangrong Shen, Yulin Xie, Qi Xu, Gang Pan, Huajin Tang, Badong Chen

Introduction

Multimodal learning aims to integrate diverse sensory inputs—such as audio and visual data—to mimic the robust perception of biological systems. However, implementing this in Spiking Neural Networks (SNNs), which are biologically inspired and energy-efficient, presents significant challenges. Specifically, SNNs struggle with modality imbalance, where one sensory stream dominates the learning process, and temporal misalignment, where asynchronous data streams fail to synchronize effectively.

This paper introduces the Temporal Attention-Guided Adaptive Fusion (TAAF) framework. By combining a dynamic temporal attention mechanism with an adaptive loss function, the proposed architecture achieves state-of-the-art performance on audio-visual tasks while maintaining the inherent energy efficiency of neuromorphic computing.

The Challenges of Multimodal SNNs

Traditional Artificial Neural Networks (ANNs) process continuous values, but SNNs communicate via discrete "spikes" over time. This temporal nature introduces two primary hurdles:

  1. Modality Imbalance: During training, different modalities often converge at different rates. For instance, in emotion recognition, audio features might be more easily discriminative than visual ones. If the audio branch optimizes too quickly, the network may ignore the visual branch, leading to a suboptimal joint representation.
  2. Temporal Misalignment: Real-world sensory data is rarely perfectly synchronized. A visual event might precede its associated sound, or sensors may operate at different sampling frequencies. Static fusion methods, which apply fixed weights to modalities, fail to capture these time-varying interactions.

Methodology

To address these issues, the researchers developed a three-part strategy to harmonize spiking data streams:

1. Temporal Attention-guided Adaptive Fusion (TAAF) Module

Rather than assigning static weights to each modality, the TAAF module calculates an "importance score" for fused features at every individual timestep. Utilizing learnable query and key projections—similar to the attention mechanisms found in Transformers—the module identifies which moments in time are most critical for the task, allowing the network to dynamically shift its focus between audio and visual cues as the sequence unfolds.

2. Temporal Adaptive Balanced Fusion Loss

This novel loss function acts as a regulatory mechanism during training. It utilizes the attention scores generated by the TAAF module to modulate the learning rates of specific modalities. If one modality begins to dominate the optimization process, the loss function suppresses its gradient, effectively "slowing down" the leading modality to allow the underperforming one to catch up. This ensures a more balanced and robust integration of features.

3. Learnable Time-Warping

To reconcile the different "clocks" of heterogeneous sensors, the framework employs convolutional time-alignment modules. These modules synchronize disparate spike trains into a unified temporal resolution, ensuring that the TAAF module receives temporally coherent data.

Experimental Results

The TAAF-SNN framework was evaluated on three major benchmarks: CREMA-D (speech emotion recognition), AVE (Audio-Visual Event localization), and EAD (Event-Audio Digit classification).

Performance and Accuracy

The proposed model achieved state-of-the-art results, outperforming both existing SNN architectures and several traditional ANN baselines:

  • CREMA-D: 77.55% accuracy.
  • AVE: 70.65% accuracy.

These results were achieved using only 3 to 4 timesteps, demonstrating that the model can make highly accurate decisions with minimal temporal depth.

Energy Efficiency

A core advantage of the SNN approach is its computational sparsity. On the CREMA-D dataset, the TAAF-SNN consumed an estimated 18.5 MJ, compared to 49.91 MJ for an ANN with a comparable architecture. This represents a 63% reduction in energy consumption, highlighting the potential for deploying these models on power-constrained edge devices.

Discussion and Future Work

The success of the TAAF-SNN lies in its ability to mimic cortical multisensory integration, where the brain prioritizes inputs based on temporal congruence and relevance. By moving away from static fusion, the model captures "temporal salience," such as prioritizing visual motion at the onset of an action and auditory cues during its conclusion.

While the current framework excels in short temporal windows, future research will explore scaling these mechanisms to longer sequences and deploying them on physical neuromorphic hardware like Intel’s Loihi. The authors also envision applying this framework to neurorobotics, where real-time, energy-efficient multisensory integration is critical for navigating complex, unpredictable environments.

Summary updated

BibTeX
@inproceedings{shen2025spiking,
  author = {Jiangrong Shen and Yulin Xie and Qi Xu and Gang Pan and Huajin Tang and Badong Chen},
  title = {{Spiking Neural Networks with Temporal Attention-Guided Adaptive Fusion for imbalanced Multi-modal Learning}},
  booktitle = {ACM Multimedia},
  year = {2025},
  doi = {10.1145/3746027.3755622},
}
Figure