Multi-Timescale Motion-Decoupled Spiking Transformer for Audio-Visual Zero-Shot Learning

Wenrui Li, Penghong Wang, Xingtao Wang, Wangmeng Zuo, Xiaopeng Fan, Yonghong Tian

Abstract

Audio-visual zero-shot learning (ZSL) has been extensively researched for its capability to classify video data from unseen classes during training. Nevertheless, current methodologies often struggle with background scene biases and inadequate motion detail. This paper proposes a novel dual-stream Multi-Timescale Motion-Decoupled Spiking Transformer (MDST++), which decouples contextual semantic information and sparse dynamic motion information. The recurrent joint learning unit is proposed to extract contextual semantic information and capture joint knowledge across various modalities to understand the environment of actions. By converting RGB images to events, our method captures motion information more accurately and mitigates background scene biases. Moreover, we introduce a discrepancy analysis block to model audio motion information. To enhance the robustness of SNNs in extracting temporal and motion cues, we dynamically adjust the threshold of Leaky Integrate-and-Fire neurons based on global motion and contextual semantic information. Our experiments validate the effectiveness of MDST++, demonstrating their consistent superiority over state-of-the-art methods on mainstream benchmarks. Additionally, incorporating motion and multi-timescale information significantly improves HM and ZSL accuracy by 26.2% and 39.9%.

Introduction: The Challenge of Background Bias

In Audio-Visual Zero-Shot Learning (AV-ZSL), models must identify actions in videos (e.g., "basketball dunk") that were not present in the training set, relying instead on semantic descriptions. Current models face two major hurdles: Background Scene Bias, where models rely on environmental cues (like a stadium) rather than the action itself, and Inadequate Motion Detail, where dense RGB frames obscure subtle temporal cues.

Figure 1: The Event Generation Model (EGM) converts RGB frames into sparse events, effectively denoising static backgrounds (e.g., grass) to highlight motion.

The authors argue that true zero-shot generalization requires decoupling contextual semantics (the "where") from dynamic motion (the "what is happening"). By explicitly separating these streams, the model becomes robust to "scene leaks" where it might otherwise guess an action based on the background rather than the actor's movements.

Methodology: The MDST++ Architecture

The proposed MDST++ architecture employs a dual-stream approach to handle motion and context independently:

  • Event Generation Model (EGM): Instead of raw RGB frames, the model uses synthetic events that mimic neuromorphic cameras. This process naturally "denoises" the video, stripping away static backgrounds and highlighting only moving objects.
  • Spiking Neural Networks (SNNs) & SpikeFormer: These events are processed by SNNs, which communicate via discrete "spikes." A Spiking Transformer (SpikeFormer) uses self-attention to capture long-range temporal dependencies within these spike trains.
  • Recurrent Joint Learning Unit (RJLU): This unit handles the context stream, fusing audio and visual features over time to build a stable understanding of the environment.
  • Multi-Stage Timestep Shrinkage: Temporal resolution is gradually reduced as data moves deeper into the network. Initial layers capture fine-grained motion, while deeper layers focus on high-level semantic reasoning.
  • Dynamic Threshold Adjustment: The firing thresholds of neurons are dynamically adjusted based on scene complexity and motion intensity, preventing noise from over-exciting the network.
Figure 2: Overview of the MDST++ framework, illustrating the dual-stream processing of video and audio through RJLU and SNN components.

Experimental Results

MDST++ was evaluated on three mainstream benchmarks: VGGSound-GZSL, UCF101-GZSL, and ActivityNet-GZSL. The model consistently outperformed state-of-the-art (SOTA) methods:

  • UCF101: Achieved a Harmonic Mean (HM) of 33.38%, surpassing previous SOTA methods like TCaF (31.72%).
  • ActivityNet: Improved ZSL accuracy to 14.12%, up from the previous best of 12.55%.
  • Ablation Studies: Removing the motion-modeling branch caused a substantial performance drop, proving that motion decoupling is the primary driver of its zero-shot capability. Overall, the inclusion of motion and multi-timescale information improved ZSL accuracy by up to 39.9%.

Conclusion

MDST++ demonstrates that decoupling motion from static context is essential for robust audio-visual zero-shot learning. By utilizing synthetic events and SNNs, the model effectively mitigates background bias and captures high-frequency motion details. While the current model uses synthetic events, future work will focus on leveraging real neuromorphic hardware and further optimizing the integration between motion and context streams.

Summary updated

Abstract

Audio-visual zero-shot learning (ZSL) has been extensively researched for its capability to classify video data from unseen classes during training. Nevertheless, current methodologies often struggle with background scene biases and inadequate motion detail. This paper proposes a novel dual-stream Multi-Timescale Motion-Decoupled Spiking Transformer (MDST++), which decouples contextual semantic information and sparse dynamic motion information. The recurrent joint learning unit is proposed to extract contextual semantic information and capture joint knowledge across various modalities to understand the environment of actions. By converting RGB images to events, our method captures motion information more accurately and mitigates background scene biases. Moreover, we introduce a discrepancy analysis block to model audio motion information. To enhance the robustness of SNNs in extracting temporal and motion cues, we dynamically adjust the threshold of Leaky Integrate-and-Fire neurons based on global motion and contextual semantic information. Our experiments validate the effectiveness of MDST++, demonstrating their consistent superiority over state-of-the-art methods on mainstream benchmarks. Additionally, incorporating motion and multi-timescale information significantly improves HM and ZSL accuracy by 26.2\% and 39.9\%.
BibTeX
@article{li2025multi,
  author = {Wenrui Li and Penghong Wang and Xingtao Wang and Wangmeng Zuo and Xiaopeng Fan and Yonghong Tian},
  title = {{Multi-Timescale Motion-Decoupled Spiking Transformer for Audio-Visual Zero-Shot Learning}},
  journal = {IEEE Transactions on Circuits and Systems for Video Technology},
  year = {2025},
  doi = {10.1109/tcsvt.2025.3574499},
}
Figure