Spiking neural networks (SNNs) pass information between layers as binary pulses over a few discrete time steps, so on neuromorphic chips most multiplications become sparse additions. Their accuracy still trails ordinary networks, and that shortfall has been measured almost entirely on image classification. In audio, every spiking model we found is a single-domain specialist: one handles environmental tagging, another handles spoken keywords. None has been tested as a general-purpose audio encoder whose frozen output must serve speech, music and environmental tasks at once. This note proposes building the first such encoder and using it to answer a question that image benchmarks cannot pose: is the spiking accuracy gap uniform, or does it open mainly on tasks that need fine pitch and timing detail?
Our hypothesis is that the gap is uneven. Coarse spectral envelopes, which are enough to recognize a dog bark, should survive conversion to sparse spikes. Fine harmonic and timing structure, which is needed to name a musical pitch, should be destroyed by spiking neurons that all share one decay rate and are read out as spike counts. If that is the cause, a cheap change native to spiking hardware should close the music gap selectively without hurting the other domains. The change is to give each channel its own learned membrane decay rate, so that a fast-forgetting population tracks onsets and pitch while a slow-forgetting population integrates toward invariant categories. This works like the two-speed "Slowfast" design from universal audio ANNs, but the two speeds come from the neurons' own time constants rather than from duplicated convolution streams.
Core Claim
We claim three things. First, a compact spiking student distilled from a frozen general-purpose audio teacher will show a domain-dependent gap under frozen-embedding evaluation, relative to an architecture-matched non-spiking student trained identically: small on environmental tasks, large on pitch and fine-timing music tasks. Second, most of that excess music gap comes from two sources. One is temporal homogeneity, meaning a single shared decay constant. The other is low-precision readout, meaning embeddings formed from spike counts with only $T$ possible levels. Third, per-channel learnable decay combined with a membrane-potential readout recovers most of the music gap at nearly unchanged synaptic-operation cost. What would be new is the per-domain diagnosis of where the spiking gap lives, and a fix aimed at the diagnosed cause rather than at average accuracy.
Positioning Against Prior Work
Three groups of papers play different roles here.
The first group motivates the gap. Two spiking audio systems are confined to one domain each. SATRN builds an event-driven spiking tagger with adaptive time chunking and evaluates only on UrbanSound8K and FSD50K, with supervised training. SpikeSCR builds a spiking transformer for keyword spotting and distills from long to short simulation windows, but only within speech datasets. S-CMRL and STFT do cross-modal spiking work, but they lean on video frames or continuous text embeddings. Neither produces a standalone audio encoder. We differ from all four in the same way: one audio-only spiking encoder, with no task labels during pretraining, judged on frozen transfer across all three domains.
The second group supplies the evaluation and the domain conflict. HEAR standardizes frozen-embedding evaluation over 19 tasks and documents the blind spots: speech models fail at pitch, pitch models fail at speech semantics. Towards Learning Universal Audio Representations introduces the HARES suite and makes four relevant points. Supervised AudioSet accuracy does not predict universal transfer. Contrastive pretraining transfers better. Two temporal resolutions help. Batch normalization (re-centering and re-scaling activations using batch statistics) hurts cross-domain transfer.
That last finding collides with the spiking literature. Spikformer reports that batch normalization is needed for stable spiking training. RTFormer builds on a time-sliding variant of it and folds its parameters into firing thresholds. We treat this contradiction as a controlled ablation rather than assuming either side is right.
The third group supplies methods. EnOF-SNN shows that SNN accuracy is limited by impoverished final-layer features and that replacing the last spiking layer with a continuous activation helps; we reuse that idea as a readout ablation. TSER shows that distilling at every time step beats averaging over time first. We carry this over from image logits to frame-level audio embeddings, where the time steps are real audio time rather than repeated copies of a static input. OpenBEATs provides the open, multi-domain teacher.
The dependence on a large teacher is a known limitation of distillation-based SNN training. We accept it, because the student is what gets deployed.
Research Program
The student is a small spiking convolution-plus-attention encoder of roughly the size of the 5.3M-parameter network in DeLoRes. It uses leaky integrate-and-fire neurons (units that accumulate input, leak a fraction each step, and emit a spike when they cross a threshold) and is trained with surrogate gradients. Unlike image spiking transformers, which repeat a static input for $T$ steps, each spiking time step consumes the next log-mel frame. The neuron leak therefore performs genuine temporal integration over the audio.
The key ingredient is per-channel decay. Each channel $c$ in a layer has a membrane update $U_c[t] = \lambda_c U_c[t-1] + I_c[t] - V_{\text{th}} S_c[t-1]$, where $U_c$ is the membrane potential, $I_c$ the input current, $S_c$ the binary spike and $V_{\text{th}}$ the threshold. The decay $\lambda_c \in (0,1)$ is learned and initialized to span a fast range and a slow range.
Training is label-free distillation from frozen OpenBEATs-Large on a multi-domain mix. AudioSet is used without labels, and FMA music and LibriSpeech speech are added so that no domain is starved. The student's last-layer membrane state at frame $t$ passes through a small linear projection $P$ and is matched to the teacher's time-aligned frame embedding $z_t$:
$$\mathcal{L} = \frac{1}{T}\sum_{t=1}^{T}\left(1 - \cos\big(P\,U[t],\, z_t\big)\right).$$
The clip embedding used for evaluation is the time-averaged membrane state. This readout costs no extra multiplications on chip, because the membrane is accumulated anyway.
Normalization is tested in two forms: standard batch normalization folded into thresholds, as in RTFormer, versus scaled weight standardization with no batch statistics, following the HARES finding.
Baselines
The naive reference is a spiking student with the same architecture, a single shared decay constant and spike-count readout, distilled identically. This is what a practitioner would build by porting a spiking transformer to audio.
The central comparison is an architecture-matched non-spiking student. It has the same layers, the same teacher, data, steps and tuning budget, with ReLU in place of spiking neurons. The gap between the spiking and ANN students on each task is the quantity the whole proposal is about. "Beating" here means shrinking the music-domain gap while keeping the environment and speech gaps no worse.
The strongest reported universal-audio result in the file is Slowfast NFNet-F0 trained with contrastive learning. It reports an average HARES score of 74.6 and 39.5 mAP on MagnaTagATune (Towards Learning Universal Audio Representations). We cite this as an upper reference, not a target for a model this small. We also measure the OpenBEATs teacher on HARES ourselves; we found no reported HARES or HEAR average for OpenBEATs on Lacuna. We likewise found no reported HEAR numeric score on Lacuna for the leading HEAR models.
For per-task context we quote supervised spiking specialists, noting that they use a different, fine-tuned protocol: SATRN reports 0.455 mAP on FSD50K and SpikeSCR reports 96.08% on Google Speech Commands v2. For a frozen-probe comparison at similar scale, DeLoRes reports 86.1% linear-probe accuracy on Speech Commands.
A supervised spiking baseline is also included: the same spiking student trained with AudioSet labels instead of teacher features. This tests whether distillation matters at all for transfer.
The ablations that isolate the key ingredients cross three choices: shared versus per-channel decay, spike-count versus membrane versus ReLU-head readout (the EnOF-style variant), and batch normalization versus weight standardization. We also add an explicit two-stream spiking Slowfast model with matched parameters. This shows whether learned decay does what duplicated pathways do, at lower cost.
All models get identical pretraining steps, data order and hyperparameter search budgets. Downstream heads follow each benchmark's fixed probe protocol.
Why not just…?
Why not just distill the teacher into a small ANN and quantize it to INT8? This is the most serious threat, and the matched ANN student is built precisely to confront it. Quantized ANNs still perform multiply-accumulates. Spiking networks replace these with sparse additions; Spikformer reports 3.96 µJ against 354.2 µJ for one attention block, though that is an estimate rather than a hardware measurement. We have no INT8 energy figures in the file and will not invent them. We will instead report accuracy against counts of each operation type and against the standard 45 nm per-operation estimates for float operations versus synaptic operations. If the INT8 ANN student dominates the spiking student's accuracy–cost frontier on every domain, we will report that plainly. The per-domain diagnosis remains useful either way.
Why not deploy three small spiking specialists behind a domain router? HEAR shows that specialists have severe cross-domain blind spots, and OpenBEATs shows that multi-domain training helps even narrow tasks such as bioacoustics. Many HEAR tasks also mix domains, for example emotional speech, so routing is ill-posed for them. We will still train three per-domain students whose combined parameter budget matches the universal student and compare total memory and per-domain accuracy. If the specialists win at equal memory, the universal-encoder premise fails for spiking hardware.
Why not just pretrain a spiking classifier on AudioSet labels? HARES shows that supervised AudioSet pretraining learns class shortcuts that discard pitch and timing, which is exactly the information whose loss we hypothesize. The supervised spiking baseline tests this directly. We expect it to be competitive on environmental tasks and worst on NSynth Pitch.
Why not just add a continuous output head, as in EnOF-SNN, and skip the decay change? A continuous head fixes readout precision but cannot restore timing information that homogeneous neurons have already smeared out in earlier layers. The readout-by-decay factorial separates the two causes. If the ReLU head alone closes the music gap, the temporal hypothesis is wrong and the contribution reduces to confirming EnOF's mechanism on audio.
Proposed Experiments and Decisive Checks
All models are evaluated with frozen embeddings on HARES, which has reported reference numbers, and on HEAR. HEAR adds the tasks most sensitive to fine pitch and timing: NSynth Pitch, MAESTRO, Mridangam and Gunshot Triangulation. Results are grouped into environment, speech and music, and the spiking-minus-ANN gap is reported per task with seeds.
The first decisive check is the gap profile of the naive spiking student. If its gap on music pitch and transcription tasks is not clearly larger than its gap on ESC-50 and FSD50K, the core hypothesis is false and the project becomes a straightforward first holistic spiking benchmark.
The second check is whether per-channel decay shrinks the music gap more than the environment gap. A further prediction follows: after training, the learned fast channels should carry most of the linear-probe information about pitch, which we test by probing fast and slow channel subsets separately.
The third check settles the normalization contradiction on cross-domain transfer.
The fourth check reports synaptic operations and firing rates per second of audio, so that every accuracy gain is shown with its cost.
As a secondary check, we repeat SATRN's below-0 dB noise test on the best student to see whether spiking robustness carries over to a universal encoder.
References
- SATRN: Spiking Audio Tagging Robust Network
- Efficient Speech Command Recognition Leveraging Spiking Neural Network and Curriculum Learning-based Knowledge Distillation
- Enhancing Audio-Visual Spiking Neural Networks through Semantic-Alignment and Cross-Modal Residual Learning
- Spiking Tucker Fusion Transformer for Audio-Visual Zero-Shot Learning
- HEAR: Holistic Evaluation of Audio Representations
- Towards Learning Universal Audio Representations
- Spikformer: When Spiking Neural Network Meets Transformer
- RTFormer: Re-parameter TSBN Spiking Transformer
- EnOF-SNN: Training Accurate Spiking Neural Networks via Enhancing the Output Feature
- Temporal Separation with Entropy Regularization for Knowledge Distillation in Spiking Neural Networks
- OpenBEATs: A Fully Open-Source General-Purpose Audio Encoder
- DeLoRes: Decorrelating Latent Spaces for Low-Resource Audio Representation Learning