SPKLIP: Aligning Spike Video Streams with Natural Language

Yongchang Gao, Meiling Jin, Zhaofei Yu, Tiejun Huang, Guozhang ChenSubmission to ICLR 2026 · Withdrawn

TL;DR

SPKLIP is the first end-to-end framework specifically designed for Spike Video-Language Alignment (Spike-VLA). By pairing a dedicated hierarchical spike feature extractor with contrastive multimodal learning, it bypasses costly intermediate frame reconstruction to directly ground ultra-high-speed spike camera streams in natural language. Across benchmarks, SPKLIP achieves a 91.15% Top-1 accuracy—massively outperforming adapted standard vision-language models—and demonstrates robust few-shot generalization on a newly collected real-world spike dataset.

What Problem Does This Paper Address?

Neuromorphic spike cameras operate at microsecond-level temporal resolutions (up to 40,000 Hz) with an ultra-wide dynamic range (~180 dB), mimicking the retinal fovea by firing binary pulses ("spikes") when pixel light accumulation exceeds a threshold. While ideal for high-speed tracking and robotics, their output is continuous, asynchronous, and extremely sparse.

State-of-the-art vision-language models (such as CLIP) excel at standard RGB video understanding but fail catastrophically when fed raw spike data due to severe modality mismatch. Prior spike-based vision techniques typically convert spike streams into pseudo-images or static frames first, throwing away fine continuous spatiotemporal dynamics. SPKLIP addresses this gap by directly aligning native spike event streams with natural language representations.

Methods

SPKLIP builds a tailored pipeline to align asynchronous spike streams with text embeddings from a Transformer (BERT-style) text encoder:

  1. Hierarchical Spike Feature Extractor (HSFE): Splits incoming spike volumes into sliding temporal blocks and processes them using Multi-Scale Temporal Filtering (MTF). Governed by a physical photon-conservation constraint, parallel convolutions use varying channel sizes: wide channels capture short-exposure, high-frequency motion (e.g., a fast ball), while narrower channels simulate longer exposures for static regions. A Spatial Attention (SA) module then weights the most salient temporal scales.
  2. SpatioTemporal Attentive Residual Network (STAR-Net): Refines HSFE representations using a modified attention-pooling backbone (MAPResNet) coupled with a temporal Transformer encoder to capture long-range action dependencies across the stream.
  3. Spike-Text Contrastive Learning (STCL): Employs symmetric InfoNCE contrastive loss over spike-video and text query embeddings to pull matching pairs together while pushing non-matching pairs apart.
  4. Full-Spiking Visual Encoder (FSVE): An exploratory, neuromorphically plausible variant that replaces conventional layers with Spiking ResNets (incorporating Leaky Integrate-and-Fire neurons and Temporal-Dependent Batch Normalization) and an energy-efficient Spiking Temporal Transformer.

Datasets

  • HMDB51-S & UCF101-S: Synthetic spike datasets converted from HMDB51 and UCF101 via high-frame-rate neural interpolation ($10\times$ to $50\times$) and biologically inspired integrate-and-fire spike encoding using the SpikeCV framework.
  • New Real-World Spike Dataset: A self-collected benchmark containing 30 high-speed, dynamic action categories (e.g., table tennis forehand loops, badminton swings) captured with an actual physical spike camera ($416 \times 250$ cropped to $320 \times 240$, 2,000–4,000 frames each), yielding 2,880 samples ($96 \times 30$).

Key Insights and Ideas

  • Naive ViT adaptation fails on spikes: Conventional Vision Transformers expect dense, spatially correlated pixel data. Feeding adapted spike features into a standard ViT frontend yielded poor convergence and sub-40% accuracy, showing that spike inputs demand native, specialized inductive biases.
  • Sim-to-Real Domain Differences: While both synthetic and real spike streams share extreme sparsity (>99% of pixels fire low spike counts), real data exhibits sparse, localized motion with sensor line artifacts, whereas synthetic data contains denser global motion and blocky compression artifacts.
  • SNN Efficiency Bottleneck Shifts to Memory: The full-spiking model (FSVE) drops computational energy by 74.1% thanks to event sparsity. However, frequent membrane potential reads/writes make memory access account for 68.99% of the SNN’s total energy, netting an overall 16.6% energy saving.
  • Context length trade-off: Ablating temporal block counts revealed that too few blocks provide insufficient temporal context, while too many cause "temporal context pollution," diluting rapid dynamic signals with surrounding static inactivity.

Key Results

  • Action Classification (HMDB51-S): SPKLIP achieved 91.15% Top-1 and 99.75% Top-5 accuracy from scratch, vastly outperforming directly adapted video-language models like Vita-CLIP (45.31% Top-1) and M2-CLIP (39.57% Top-1). On UCF101-S, SPKLIP reached 86.43% Top-1.
  • Real-World Few-Shot Transfer: Fine-tuning only the last two layers of STAR-Net on the real-world dataset yielded 55.17% (2-shot), 73.11% (4-shot), 85.78% (6-shot), and 88.12% (8-shot) Top-1 accuracy, proving strong domain transfer.
  • Text-to-Video Retrieval: On HMDB51-S, SPKLIP achieved 31.94% R@1, 63.12% R@5, and 75.10% R@10 on complex sentence-level video caption queries.

Limitations and Future Work

  • Performance Drop in Pure SNNs: While the full-spiking FSVE provides biologically plausible low-power execution, classification accuracy on UCF101-S dropped from 86.43% (ANN) to 74.14% (spiking CNN) and 67.29% (full spiking CNN + Transformer).
  • Memory Bandwidth Bottleneck: SNN energy efficiency is currently capped by SRAM memory access overhead rather than raw computation.
  • Physical Neuromorphic Deployment: Evaluations currently use simulation and CMOS 45nm analytical models; deploying SPKLIP directly onto dedicated neuromorphic edge hardware (e.g., Tianjic, Loihi) remains a target for future research.

Summary updated

Abstract

Spike cameras offer unique sensing capabilities but their sparse, asynchronous output challenges semantic understanding, especially for Spike Video-Language Alignment (Spike-VLA) where models like CLIP underperform due to modality mismatch. We introduce SPKLIP, the first architecture specifically for Spike-VLA. SPKLIP employs a hierarchical spike feature extractor that adaptively models multi-scale temporal dynamics in event streams, and uses spike-text contrastive learning to directly align spike video with language, enabling effective few-shot learning. A full-spiking visual encoder variant, integrating SNN components into our pipeline, demonstrates enhanced energy efficiency. Experiments show state-of-the-art performance on benchmark spike datasets and strong few-shot generalization on a newly contributed real-world dataset. SPKLIP's energy efficiency highlights its potential for neuromorphic deployment, advancing event-based multimodal research. The source code and dataset are available at [link removed for anonymity].
BibTeX
@misc{gao2025spklip,
  author = {Yongchang Gao and Meiling Jin and Zhaofei Yu and Tiejun Huang and Guozhang Chen},
  title = {{SPKLIP: Aligning Spike Video Streams with Natural Language}},
  year = {2025},
  eprint = {2505.12656},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.12656},
}
Figure