Spiking neural networks are models whose neurons communicate with brief all-or-nothing pulses instead of continuous numbers. On event-driven chips their energy use scales with the number of pulses and the number of active connections. That makes them attractive for always-on audio devices such as keyword spotters. These devices face a data problem: recorded audio is cheap, transcriptions are expensive. The usual remedy is pseudo-labeling. A model trained on the few labeled clips guesses labels for the unlabeled ones and then trains on its own guesses. Pseudo-labeling has a known failure mode, confirmation bias: the model memorizes its own wrong guesses and becomes more confident in them.
This note proposes a remedy that the energy budget already pays for. A spiking network that will be deployed with most of its connections removed contains two models inside one set of weights: the full dense network and its pruned, high-magnitude subnetwork. Work on pruning under label noise shows that memorized noise tends to live in small-magnitude weights. The pruned view should therefore be unable to reproduce memorized wrong labels that the dense view has absorbed. We propose to accept a pseudo-label only when the dense and pruned views of the same spiking network agree on it, and to deploy the pruned view. If this works, one mechanism filters label noise and removes most of the network's connections, without a second teacher model or an external language model.
Core Claim
In semi-supervised spiking speech models, agreement between the dense and pruned views of a single network is a better filter for wrong pseudo-labels than either model confidence or agreement across time windows. The resulting pruned student should match or beat dense semi-supervised baselines while using far fewer synaptic operations.
The time-window comparison matters because the closest prior work, SpikeMatch, does semi-supervised learning in spiking networks on images. It splits the simulation time steps into $M$ groups and accepts a pseudo-label when $M-1$ groups agree. That works for a static image presented repeatedly. It is poorly suited to speech, where early and late time steps hear different sounds, so disagreement between the first and second half of a word is expected rather than a sign of noise. SpikeMatch also uses no weight sparsity. Our proposal replaces agreement across time with agreement across sparsity levels. That axis stays meaningful for time-varying signals, and the deployed model inherits the connection savings.
What the Prior Work Supplies and What It Rules Out
The noise-filtering premise comes from two lines of pruning research on ordinary networks. Why Does Pruning during Training Work? analyzes two-layer convolutional networks. It proves that dynamic top-K magnitude pruning mutes weights aligned with noise because those weights stay small. On CIFAR-100 with 40% label noise, keeping 80% of the weights gave a 1.48% accuracy gain over dense training. Sparse Double Descent is the constraint on our design. It finds that under label noise, moderate pruning can make overfitting worse (a "critical phase") before high sparsity helps (a "sweet phase"), and that more noise moves the harmful zone. We therefore cannot pick a sparsity level on faith. The program must measure where these phases fall when the noise comes from pseudo-labels rather than injected flips. Neither paper studies pseudo-labeling, speech, or spiking networks.
The efficiency side builds on Weight Sparsity Complements Activity Sparsity in Neuromorphic Language Models. That paper shows in event-based recurrent units that connection sparsity and firing sparsity multiply: a 95%-pruned model needs 0.5M operations against 20.2M for a dense LSTM, with perplexity 65.2 on Penn Treebank. Its setting is fully supervised text and post-training pruning. Gradient Rewiring supplies a training-time pruning method for spiking networks that lets removed connections regrow. It keeps 92.03% CIFAR-10 accuracy at about 5% connectivity and also lowers firing rates, but only under full supervision. The audio models come from Global-Local Convolution for Energy-efficient Keyword Spotting, a raw-waveform spiking keyword spotter trained fully supervised, and from the Heidelberg spiking datasets, which provide spiking speech benchmarks and recurrent spiking baselines.
On the speech side, Iterative Pseudo-Labeling and Momentum Pseudo-Labeling define how dense speech models control confirmation bias: external language-model decoding in the first case, and a slowly updated averaged teacher in the second. Both are the reference recipes we must compete with, even though neither considers sparsity or spiking.
Mechanism
Let $w$ be the network's weights and $m_s(w)$ a mask that keeps the largest-magnitude fraction $1-s$ of weights, recomputed during training as in dynamic top-K pruning. For an unlabeled clip $u$ under weak augmentation, let $p_d(u)$ and $p_s(u)$ be the class distributions from the dense network and from the masked network $w \odot m_s(w)$. Both are read out from spike counts accumulated over the clip. The pseudo-label is $\hat y = \arg\max p_s(u)$. It is accepted only if
$$\arg\max p_d(u) = \arg\max p_s(u) \quad \text{and} \quad \max p_s(u) > \tau,$$
where $\tau$ is a confidence threshold. Accepted clips train both views on strongly augmented audio, a SpecAugment-style asymmetric scheme, with a shared-weight loss. Labeled clips train both views as usual. Gradients from the pruned view flow only through kept weights, and Gradient Rewiring lets pruned connections re-enter.
The dense view is where memorization shows up. The pruned view cannot follow it into noise, so their disagreement is the noise signal. We also plan to anneal $s$ upward as pseudo-label error falls, and to calibrate the target $s$ against the sparsity curve so that training avoids the critical phase. The deployed model is the pruned view alone.
Research Program
The work has three parts. The first is diagnostic. On labeled audio with injected label noise at rates matching measured pseudo-label error, we measure how well dense–pruned disagreement flags corrupted labels, as area under the ROC curve. We compare it to low confidence and to temporal-window disagreement. This tests the mechanism before any semi-supervised claim.
The second part is the semi-supervised system on spiking keyword spotting and spiking speech classification. It includes a full sparsity sweep from 0% to 99% to map whether the double-descent phases appear under pseudo-label noise and where.
The third part is energy accounting. We count synaptic operations as spikes multiplied by nonzero fan-out, following the multiplicative accounting of the language-model paper. These are estimates, not measurements on neuromorphic hardware, and will be reported as such.
The data are Google Speech Commands V2 (35 classes) and Spiking Speech Commands, each with 1%, 5%, and 10% of training labels kept and the rest treated as untranscribed. Spiking Heidelberg Digits provides a domain-shift split: English digits labeled, German digits unlabeled, evaluated on German. The shift is where confirmation bias bites hardest. The backbone is the raw-waveform spiking keyword spotter for Speech Commands and a recurrent spiking network for the Heidelberg sets. A stretch experiment tests a small spiking model trained with CTC, a loss that aligns frame outputs to transcripts, on LibriSpeech 100h labeled plus 360h unlabeled, scored by word-error recovery rate. We treat it as exploratory because the Heidelberg paper and the neuromorphic language-model paper both document that spiking models still trail dense ones on sequence tasks.
Baselines
The naive reference is the same spiking backbone trained only on the labeled subset, dense and at the target sparsity. Every semi-supervised gain is measured from it. For context, fully supervised reference points exist but are not semi-supervised competitors. The raw-waveform spiking keyword spotter reports 93.0% accuracy on Speech Commands V1 12-class with 70.1K parameters and an 8.3% firing rate (Global-Local Convolution). The Heidelberg paper reports 83.2% for an optimized recurrent spiking network on SHD and 50.9% on SSC.
The strongest direct competitor is SpikeMatch-style temporal agreement ported to audio: the same backbone, the same asymmetric augmentation, and the same labeled/unlabeled split, with acceptance by $M-1$ of $M$ time-window groups. It is the only published semi-supervised spiking method. We found no reported result on Lacuna for SpikeMatch or any semi-supervised spiking model on speech, so we run it ourselves. The second strongest is a momentum teacher in the style of Momentum Pseudo-Labeling, applied to the dense spiking network. That paper reports 81.2% word-error recovery rate on LibriSpeech 100h/360h, against 54.5% for plain pseudo-labeling without a language model, for dense non-spiking models. This number is our target only in the LibriSpeech stretch experiment. We also run a dense confidence-threshold baseline (FixMatch-style), and a dense artificial network trained with momentum pseudo-labeling, then pruned and converted to spiking form.
Two ablations isolate the key ingredient. The first applies pruning as a regularizer with confidence filtering only, without the agreement test. It separates "sparsity regularizes" from "sparsity-view agreement filters." The second is a dense spiking network shrunk to the same number of nonzero weights, which separates structured noise filtering from plain capacity reduction.
All methods get the same augmentations, the same number of training steps, and the same hyperparameter search of 20 trials per method on a held-out labeled validation set. Results use three seeds. Beating the field means the pruned-view model exceeds the best of temporal agreement and the spiking momentum teacher by more than seed variance at the 1% and 5% label budgets, at no more than a fifth of their synaptic operations.
Why not just…?
Why not just use confidence thresholding and strong augmentation? These are the standard tools, and on dense speech models with a language model they reach 7.11% test-other word error rate on LibriSpeech 100h (Iterative Pseudo-Labeling). However, confidence goes up precisely on examples the model has memorized, which is why it fails as a confirmation-bias signal. SpikeMatch reports that FixMatch-style methods transfer poorly to spiking networks. Our confidence-only baseline, run under the same augmentation, measures this directly. If it matches our method, the agreement test adds nothing and we will say so.
Why not just train a smaller dense model or add weight decay? Plain capacity reduction is the most serious threat, because the sweet-phase gains in Sparse Double Descent could be a capacity effect. The matched-nonzero dense control and the pruning-without-agreement ablation settle it. The mechanism predicts that the gain comes from the dense–pruned comparison, which a small dense model does not have. If the small model ties, the claim reduces to "sparse spiking pseudo-labeling is as good as small dense," which is still useful for energy but not a regularization result.
Why not train a dense network with momentum pseudo-labeling, then prune and convert it? This uses mature tools and may well reach higher accuracy. Two costs work against it. Conversion to spiking form needs long simulation windows. Post-hoc pruning cannot revive removed connections, the failure Gradient Rewiring was designed to fix. The pipeline also never lets sparsity act on pseudo-label noise during training. We run it at matched operations, and it is the baseline most likely to win on accuracy at 10% labels. Our claim is limited to the low-label and domain-shift regimes.
Why not use a momentum teacher instead of the pruned view? An averaged teacher stabilizes labels but does not target memorization in small weights, and it doubles weight storage during training. The two are compatible. We therefore also test the combination, with the pruned view of the averaged teacher producing labels. The combination winning would sharpen the claim rather than refute it.
Decisive Checks
The idea stands or falls on four measurements. First, on injected-noise audio, dense–pruned disagreement must detect wrong labels better than confidence and temporal disagreement. Second, at equal acceptance rates, its accepted pseudo-labels must be more accurate than those of both alternatives. Third, the sparsity sweep must show a range where semi-supervised accuracy exceeds the dense model. If only a critical-phase dip appears, the premise fails for pseudo-label noise. Fourth, on the English-to-German Heidelberg split, the method must beat the matched-capacity dense control, the setting where plain capacity reduction and genuine noise filtering should come apart most clearly.
References
- SpikeMatch: Semi-Supervised Learning with Temporal Dynamics of Spiking Neural Networks
- Why Does Pruning during Training Work? A Signal-to-Noise Analysis of Sparse Neural Network Training
- Sparse Double Descent: Where Network Pruning Aggravates Overfitting
- Weight Sparsity Complements Activity Sparsity in Neuromorphic Language Models
- Pruning of Deep Spiking Neural Networks through Gradient Rewiring
- Global-Local Convolution with Spiking Neural Networks for Energy-efficient Keyword Spotting
- The Heidelberg spiking datasets for the systematic evaluation of spiking neural networks
- Iterative Pseudo-Labeling for Speech Recognition
- Momentum Pseudo-Labeling for Semi-Supervised Speech Recognition