TL;DR
SPKLIP is the first end-to-end framework specifically designed for Spike Video-Language Alignment (Spike-VLA). By pairing a dedicated hierarchical spike feature extractor with contrastive multimodal learning, it bypasses costly intermediate frame reconstruction to directly ground ultra-high-speed spike camera streams in natural language. Across benchmarks, SPKLIP achieves a 91.15% Top-1 accuracy—massively outperforming adapted standard vision-language models—and demonstrates robust few-shot generalization on a newly collected real-world spike dataset.
What Problem Does This Paper Address?
Neuromorphic spike cameras operate at microsecond-level temporal resolutions (up to 40,000 Hz) with an ultra-wide dynamic range (~180 dB), mimicking the retinal fovea by firing binary pulses ("spikes") when pixel light accumulation exceeds a threshold. While ideal for high-speed tracking and robotics, their output is continuous, asynchronous, and extremely sparse.
State-of-the-art vision-language models (such as CLIP) excel at standard RGB video understanding but fail catastrophically when fed raw spike data due to severe modality mismatch. Prior spike-based vision techniques typically convert spike streams into pseudo-images or static frames first, throwing away fine continuous spatiotemporal dynamics. SPKLIP addresses this gap by directly aligning native spike event streams with natural language representations.
Methods
SPKLIP builds a tailored pipeline to align asynchronous spike streams with text embeddings from a Transformer (BERT-style) text encoder:
- Hierarchical Spike Feature Extractor (HSFE): Splits incoming spike volumes into sliding temporal blocks and processes them using Multi-Scale Temporal Filtering (MTF). Governed by a physical photon-conservation constraint, parallel convolutions use varying channel sizes: wide channels capture short-exposure, high-frequency motion (e.g., a fast ball), while narrower channels simulate longer exposures for static regions. A Spatial Attention (SA) module then weights the most salient temporal scales.
- SpatioTemporal Attentive Residual Network (STAR-Net): Refines HSFE representations using a modified attention-pooling backbone (MAPResNet) coupled with a temporal Transformer encoder to capture long-range action dependencies across the stream.
- Spike-Text Contrastive Learning (STCL): Employs symmetric InfoNCE contrastive loss over spike-video and text query embeddings to pull matching pairs together while pushing non-matching pairs apart.
- Full-Spiking Visual Encoder (FSVE): An exploratory, neuromorphically plausible variant that replaces conventional layers with Spiking ResNets (incorporating Leaky Integrate-and-Fire neurons and Temporal-Dependent Batch Normalization) and an energy-efficient Spiking Temporal Transformer.
Datasets
- HMDB51-S & UCF101-S: Synthetic spike datasets converted from HMDB51 and UCF101 via high-frame-rate neural interpolation ($10\times$ to $50\times$) and biologically inspired integrate-and-fire spike encoding using the SpikeCV framework.
- New Real-World Spike Dataset: A self-collected benchmark containing 30 high-speed, dynamic action categories (e.g., table tennis forehand loops, badminton swings) captured with an actual physical spike camera ($416 \times 250$ cropped to $320 \times 240$, 2,000–4,000 frames each), yielding 2,880 samples ($96 \times 30$).
Key Insights and Ideas
- Naive ViT adaptation fails on spikes: Conventional Vision Transformers expect dense, spatially correlated pixel data. Feeding adapted spike features into a standard ViT frontend yielded poor convergence and sub-40% accuracy, showing that spike inputs demand native, specialized inductive biases.
- Sim-to-Real Domain Differences: While both synthetic and real spike streams share extreme sparsity (>99% of pixels fire low spike counts), real data exhibits sparse, localized motion with sensor line artifacts, whereas synthetic data contains denser global motion and blocky compression artifacts.
- SNN Efficiency Bottleneck Shifts to Memory: The full-spiking model (FSVE) drops computational energy by 74.1% thanks to event sparsity. However, frequent membrane potential reads/writes make memory access account for 68.99% of the SNN’s total energy, netting an overall 16.6% energy saving.
- Context length trade-off: Ablating temporal block counts revealed that too few blocks provide insufficient temporal context, while too many cause "temporal context pollution," diluting rapid dynamic signals with surrounding static inactivity.
Key Results
- Action Classification (HMDB51-S): SPKLIP achieved 91.15% Top-1 and 99.75% Top-5 accuracy from scratch, vastly outperforming directly adapted video-language models like Vita-CLIP (45.31% Top-1) and M2-CLIP (39.57% Top-1). On UCF101-S, SPKLIP reached 86.43% Top-1.
- Real-World Few-Shot Transfer: Fine-tuning only the last two layers of STAR-Net on the real-world dataset yielded 55.17% (2-shot), 73.11% (4-shot), 85.78% (6-shot), and 88.12% (8-shot) Top-1 accuracy, proving strong domain transfer.
- Text-to-Video Retrieval: On HMDB51-S, SPKLIP achieved 31.94% R@1, 63.12% R@5, and 75.10% R@10 on complex sentence-level video caption queries.
Limitations and Future Work
- Performance Drop in Pure SNNs: While the full-spiking FSVE provides biologically plausible low-power execution, classification accuracy on UCF101-S dropped from 86.43% (ANN) to 74.14% (spiking CNN) and 67.29% (full spiking CNN + Transformer).
- Memory Bandwidth Bottleneck: SNN energy efficiency is currently capped by SRAM memory access overhead rather than raw computation.
- Physical Neuromorphic Deployment: Evaluations currently use simulation and CMOS 45nm analytical models; deploying SPKLIP directly onto dedicated neuromorphic edge hardware (e.g., Tianjic, Loihi) remains a target for future research.
Summary updated