Multi-Timescale Motion-Decoupled Spiking Transformer for Audio-Visual Zero-Shot Learning

Wenrui Li, Penghong Wang, Xingtao Wang, Wangmeng Zuo, Xiaopeng Fan, Yonghong Tian

TL;DR

The Multi-Timescale Motion-Decoupled Spiking Transformer (MDST++) tackles the "background bias" problem in audio-visual zero-shot learning by separating static scene information from dynamic motion. By converting video into synthetic "events" and processing them with biologically-inspired Spiking Neural Networks (SNNs), the model achieves state-of-the-art performance, improving accuracy by up to 39.9% on major benchmarks.

What Problem Does This Paper Address?

In Audio-Visual Zero-Shot Learning (AV-ZSL), models are tasked with identifying actions in videos (like "basketball dunk") that they never saw during training, relying instead on semantic descriptions.

Current models face two major hurdles:

  1. Background Scene Bias: Models often cheat by looking at the environment rather than the action. For example, if a model sees a running track, it might classify the activity as "running" even if the person is actually "baby sleeping."
  2. Inadequate Motion Detail: Standard RGB frames are dense with static information, making it hard for models to focus on the subtle, high-frequency temporal cues that define an action.

The authors argue that to truly "understand" an action for zero-shot generalization, a model must decouple contextual semantics (the "where") from dynamic motion (the "what is happening").

Methods

The researchers propose a dual-stream architecture called MDST++. Its core innovation lies in how it handles motion and time:

  • Event Generation Model (EGM): Instead of processing raw RGB frames, the model converts them into synthetic events. This mimics neuromorphic (event-based) cameras that only record changes in brightness. This process naturally "denoises" the video, stripping away static backgrounds (like the grass in a baseball game) and highlighting only the moving players and objects.
  • Spiking Neural Networks (SNNs) & SpikeFormer: To process these sparse events, the authors use SNNs, which communicate via discrete "spikes" rather than continuous values. They integrate a Spiking Transformer (SpikeFormer) that uses self-attention to capture long-range temporal dependencies within these spike trains.
  • Multi-Stage Timestep Shrinkage: A novel strategy where the temporal resolution (timesteps) is gradually reduced as the data moves deeper into the network. This allows the initial layers to capture fine-grained motion while deeper layers focus on high-level semantic reasoning, reducing computational latency.
  • Recurrent Joint Learning Unit (RJLU): This unit handles the "context" stream, fusing audio and visual features over time to build a stable understanding of the environment.
  • Dynamic Threshold Adjustment: Biological neurons have firing thresholds. MDST++ dynamically adjusts these thresholds based on the complexity of the scene and the intensity of the motion, preventing the network from becoming over-excited by noise or "dying" due to inactivity.

Datasets

The model was evaluated on three mainstream audio-visual benchmarks:

  • VGGSound-GZSL: A massive dataset with over 300 sound categories.
  • UCF101-GZSL: A classic action recognition dataset with 101 classes.
  • ActivityNet-GZSL: A large-scale dataset focusing on complex daily activities.

The authors used "Generalized" Zero-Shot Learning (GZSL) settings, meaning the model is tested on its ability to recognize both seen and unseen classes simultaneously—a much harder and more realistic task.

Key Insights and Ideas

  • Decoupling is Essential: The "aha moment" is that motion and scene are often confounded. By explicitly separating them into two streams, the model becomes more robust to "scene leaks" where it guesses the action based on the stadium rather than the athlete.
  • Events as a Denoising Tool: Converting RGB to events is a powerful inductive bias. It forces the model to ignore the 90% of pixels that don't change, effectively "denoising" the background for free.
  • SNNs are More Than Just Efficient: While SNNs are often touted for low-power hardware, this paper shows they are naturally suited for event-based data, capturing the "asynchronous" nature of motion better than standard artificial neural networks.
  • The Power of Timestep Shrinkage: Gradually reducing timesteps acts like a temporal pyramid, allowing the model to "zoom out" from individual movements to the overall action context.

Key Results

MDST++ consistently outperformed state-of-the-art (SOTA) methods across all benchmarks:

  • Accuracy Boost: On the UCF101 dataset, MDST++ achieved a Harmonic Mean (HM) of 33.38%, significantly higher than previous SOTA methods like TCaF (31.72%) and the original MDST (31.36%).
  • Zero-Shot Generalization: On ActivityNet, the model improved ZSL accuracy to 14.12%, a notable jump from the previous best of 12.55%.
  • Ablation Success: Removing the motion-modeling (MIM) branch caused a "substantial performance drop" in unseen classes, proving that motion decoupling is the primary driver of its zero-shot capability.

Limitations and Future Work

  • Synthetic vs. Real Events: The model uses synthetic events generated from RGB video. While effective, it doesn't fully leverage the microsecond-level temporal resolution of actual neuromorphic hardware.
  • Computational Complexity: While SNNs are theoretically efficient, training them on standard GPUs (which are optimized for dense matrices, not sparse spikes) remains a challenge.
  • Integration: The authors note that future work will focus on better integrating the features between the SNN (motion) and RJLU (context) streams to handle even more complex, multi-actor scenarios.

Summary updated

Abstract

Audio-visual zero-shot learning (ZSL) has been extensively researched for its capability to classify video data from unseen classes during training. Nevertheless, current methodologies often struggle with background scene biases and inadequate motion detail. This paper proposes a novel dual-stream Multi-Timescale Motion-Decoupled Spiking Transformer (MDST++), which decouples contextual semantic information and sparse dynamic motion information. The recurrent joint learning unit is proposed to extract contextual semantic information and capture joint knowledge across various modalities to understand the environment of actions. By converting RGB images to events, our method captures motion information more accurately and mitigates background scene biases. Moreover, we introduce a discrepancy analysis block to model audio motion information. To enhance the robustness of SNNs in extracting temporal and motion cues, we dynamically adjust the threshold of Leaky Integrate-and-Fire neurons based on global motion and contextual semantic information. Our experiments validate the effectiveness of MDST++, demonstrating their consistent superiority over state-of-the-art methods on mainstream benchmarks. Additionally, incorporating motion and multi-timescale information significantly improves HM and ZSL accuracy by 26.2% and 39.9%.
BibTeX
@article{li2025multi,
  author = {Wenrui Li and Penghong Wang and Xingtao Wang and Wangmeng Zuo and Xiaopeng Fan and Yonghong Tian},
  title = {{Multi-Timescale Motion-Decoupled Spiking Transformer for Audio-Visual Zero-Shot Learning}},
  journal = {IEEE Transactions on Circuits and Systems for Video Technology},
  year = {2025},
  doi = {10.1109/tcsvt.2025.3574499},
}

Loading related content...

Figure