Voice commands for smart homes and hearables are usually handled in two stages. A speech recognizer first writes out the words, and a second model then extracts the request, for example "action: turn off, object: lights, location: kitchen." Systems that map audio straight to that request, called end-to-end spoken language understanding (SLU), avoid compounding transcription errors. Today, though, they run on dense floating-point networks that do the same work for every audio frame, including silence and filler words. Spiking neural networks (SNNs) take a different approach. Their neurons send occasional binary pulses instead of computing a number at every step, so a silent neuron costs almost nothing. Speech-to-intent looks like a natural fit because most of an utterance carries no meaning for the task. Prior work found that direct speech-to-intent models learn to attend mostly to the few stretches of audio containing the verb and its arguments, and to ignore phrases like "can you please" (Serdyuk et al.).
We propose a spiking speech-to-intent network trained by layer-wise distillation from a pretrained phoneme→word→intent model. It carries an activity penalty weighted by how much each moment of audio matters to the teacher's decision. The claim has two parts. First, distillation from a pretrained hierarchy closes most of the accuracy gap that spiking models trained from scratch will suffer on small SLU datasets. Second, with meaning-weighted sparsity, the number of spikes in the upper layers grows with the number of meaning-bearing words in an utterance rather than with its duration. If the second part holds, the energy argument for neuromorphic SLU rests on a measured property of the task, not just the generic claim that additions are cheaper than multiplications.
What exists and what is missing
Spiking networks already do well on isolated speech classification. A dilated convolutional SNN with leaky integrate-and-fire neurons reaches 5.5% error on 12-class Google Speech Commands with 0.13M parameters and under 5% firing activity (Low-Activity Supervised Convolutional SNNs). A leaky integrate-and-fire neuron sums its inputs into an internal voltage that slowly decays, and it pulses when that voltage crosses a threshold. The same paper shows the leak matters: removing it pushes error above 12%. A surrogate-gradient baseline with adaptive neurons reaches 94.62% on Spiking Heidelberg Digits (Bittar & Garner-style surrogate baseline). A surrogate gradient is a smooth stand-in for the derivative of the non-differentiable spike. Both works classify single words or digits, not compositional commands.
Spiking acoustic models for large-vocabulary recognition reach 10.1% word error on LibriSpeech-100h against 9.5% for the matched ANN (Wu et al.), but they sit inside a transcription pipeline. On the language side, spiking text models distill BERT (SpikingBERT) or sparsify matrix-multiplication-free language models for Intel Loihi 2 (Richter et al.). Both take text tokens as input.
End-to-end SLU shows why pretraining is not optional here. The hierarchical model of Lugosch et al. reaches 98.8% intent accuracy on Fluent Speech Commands (FSC), against 96.6% trained from scratch. With 10% of the training data the gap widens to 97.9% versus 88.9%.
The research file reports no spiking model that maps audio directly to multi-slot intents. The obvious first attempt would take the keyword-spotting SNN and attach three classification heads for action, object, and location. We expect that to work tolerably on full FSC and to break where Lugosch's scratch model breaks: in low-data regimes and on the broader SLURP benchmark. Our proposal differs from the keyword-spotting SNNs in two ways: it transfers a pretrained linguistic hierarchy into spikes, and it shapes where spikes occur. It differs from SpikingBERT's distillation in two ways as well. It works on raw acoustic time series. It also cannot use SpikingBERT's training method, which is implicit differentiation at a steady-state firing rate. That method assumes the network settles to a fixed point on a static input, and speech does not settle. SpikingBERT therefore supplies the distillation idea but constrains the training method, which pushes us to surrogate-gradient backpropagation through time.
Core Claim
An SNN distilled layer by layer from a pretrained phoneme→word→intent teacher can match the teacher's FSC accuracy within about one point, including in the 10% data regime. With a meaning-weighted activity penalty, the upper-layer spike count per utterance becomes largely independent of added filler speech. Concretely, the student is trained with
$$\mathcal{L} = \mathcal{L}_{\text{intent}} + \lambda_{\text{KD}} \sum_{\ell} \lVert r_\ell - h_\ell \rVert^2 + \lambda_{s} \sum_{\ell, t} (1 - a_t)\, n_{\ell,t}^2 .$$
Here $\mathcal{L}_{\text{intent}}$ is the cross-entropy over the three slots. $r_\ell$ is the student's firing rate in layer $\ell$, smoothed over a short window, and $h_\ell$ is the teacher's activation in the corresponding phoneme or word layer, after a learned linear map. $n_{\ell,t}$ is the spike count in layer $\ell$ at frame $t$. $a_t \in [0,1]$ is the teacher's normalized saliency at frame $t$, computed from the gradient of the intent output with respect to the input. The squared spike penalty is the one used by Cramer-style low-activity SNNs. The new element is the weight $(1 - a_t)$, which makes spikes expensive exactly where the teacher says the audio does not matter.
Research Program
The teacher is Lugosch's architecture: a SincNet front end, a convolutional and recurrent phoneme module pretrained on LibriSpeech alignments, a word module, and an intent head. It is fine-tuned on FSC. The student mirrors this hierarchy. Its phoneme stage is a dilated spiking convolution stack with adaptive leaky neurons fed log-Mel filterbanks. Its word stage uses recurrent spiking layers. Its intent readout integrates voltage over time without spiking, following the surrogate-gradient baseline. We keep filterbank input rather than cochlear spike trains so that the front end matches the published SNN speech results. Cochlear input is a secondary ablation, because the file records an unresolved disagreement about which input suits spiking neurons.
The student is first pretrained on LibriSpeech phoneme and word targets with the distillation term. It is then fine-tuned on FSC with the full loss, using the same gradual-unfreezing schedule as the teacher.
The analysis centers on one measurement. We build controlled FSC variants that pad each test command with filler prefixes and suffixes ("hey, could you please… thanks"), using held-out speakers' recordings of the fillers spliced at word boundaries. We then plot upper-layer spikes per utterance against duration and against the number of meaning-bearing words. The prediction is a near-flat slope with duration in the word and intent layers and a steep slope in a uniformly penalized control. Because the leaky neurons reset naturally during quiet stretches, we also record the time at which the readout commits to the correct intent. This tests whether early decisions of the kind shown for streaming LSTMs by Shivakumar et al. arise here without a separate end-of-utterance head.
SLURP (SLURP) serves as a stress test for scenario and action accuracy only. Full entity filling requires generating spans, and we do not claim a spiking solution for that.
Baselines
The naive reference point is the dense teacher itself run on conventional hardware, together with Lugosch's reported numbers: 98.8% FSC intent accuracy on full data and 97.9% with 10% data (Lugosch et al.). This is the accuracy ceiling and the dense energy reference. We report its multiply-accumulate count per utterance under an identical counting script. A result counts as beating it on the efficiency question if the student stays within one point of accuracy while using at least an order of magnitude fewer estimated operation-energy units, with memory reads included.
The strongest spiking competitor is the direct transfer of the low-activity dilated SNN from speech commands (Cramer-style SNN) with three slot heads, trained from scratch on FSC. The research file reports no result for it on FSC. We give it the same filterbank input, the same parameter budget, the same number of time steps, and the same hyperparameter search budget as our student, measured in GPU-hours. Our method must beat it clearly at 10% data, where the scratch dense model is 9 points behind pretraining. It must also show lower upper-layer spikes on padded utterances.
The strongest efficiency competitor that does not use spikes is an INT8-quantized version of the teacher with ReLU activations whose zero-activation skipping is counted as free. This is the fairest version of the "just quantize it" argument. It is fine-tuned with the same data and tuning budget. We count its effective operations per utterance, crediting every exact zero.
For the cascade, we pair a spiking acoustic model in the style of Wu et al. with a SpikingBERT-style text classifier. On SLURP we compare against the reported pipeline and end-to-end figures: a best audio pipeline SLU-F1 of 70.84 and action accuracy of 81.42% on noisy transcripts (SLURP), and SpeechBrain's end-to-end SLU-F1 of 74.62 (SpeechBrain). The research file reports no spiking result on SLURP.
Three ablations isolate the key ingredients. Removing distillation tests the data-efficiency claim. Replacing the saliency weight $(1 - a_t)$ with a constant tests whether spikes are concentrated on meaning or merely reduced. Swapping adaptive neurons for non-leaky ones tests whether the natural reset is doing the segmentation.
Why not just…?
Why not just train the keyword-spotting SNN directly on FSC? FSC has only 31 intents, so this may largely work on the full dataset, and we treat it as a real threat there. The evidence against it as a general solution is Lugosch's scratch-versus-pretrained gap, which grows from 2.2 points on full data to 9 points on 10% data. Nothing about spiking suggests it escapes that gap. If the scratch SNN matches our student at 10% data and on SLURP actions, the distillation half of the proposal is refuted and we would narrow the contribution to meaning-weighted sparsity.
Why not just quantize and prune a dense model for an edge accelerator? This is the most dangerous alternative, because published spiking energy figures such as 0.9 pJ versus 4.6 pJ in 45nm CMOS (SpikingBERT) compare against 32-bit floating-point multiplies, not INT8. They also ignore memory traffic, which often dominates. Richter et al. argue that dense accelerators cannot exploit unstructured activation sparsity, but their sparse gains on Loihi 2 are projected rather than measured. We will not claim an advantage from arithmetic type alone. The INT8 zero-skipping baseline, with memory reads counted for both models, will show whether the spiking student's advantage survives a fair accounting. If it survives, it will be because meaning-weighted sparsity produces far fewer events per utterance, and the padded-utterance experiment will reveal whether that is so.
Why not just gate a dense model with a cheap speech or keyword detector? A voice-activity gate removes silence, which also reduces energy, and on clean single commands it may capture most of the savings. It cannot remove filler speech inside an utterance, which is exactly what the padded FSC test measures. We run the teacher behind an energy-based voice gate as an additional baseline. If its compute on padded utterances matches our student's spike cost, the meaning-weighted claim adds little, and we will say so.
Why not just cascade spiking recognition and spiking text understanding? SLURP already shows that transcription errors cost about five points of action accuracy (86.99% on perfect text versus 81.42% on recognized text). A cascade also runs two spiking simulations in sequence, so latency accumulates. We still run the cascade so that the comparison is measured rather than assumed.
Proposed Experiments
The decisive experiment has three parts. On FSC at 100% and 10% training data, we report intent accuracy, synaptic events per utterance, firing rate per layer, and estimated energy including memory access. We do this for the student, the scratch SNN, the INT8 teacher, and the gated teacher, with at least three seeds each and matched tuning budgets. On padded FSC, we fit the slope of upper-layer spike count against added filler duration for the saliency-weighted and uniform-penalty students. The proposal's central claim holds only if the weighted slope is well below the uniform one at equal accuracy. On SLURP, we report scenario and action accuracy against the reported cascade figures, to test whether the approach extends beyond a 31-intent task.
As a secondary check, we measure time to a correct decision on concatenated FSC utterances to see whether the neuron leak alone prevents the long-stream degradation that Shivakumar et al. observed in offline models. If Loihi 2 access is available, we will deploy the FSC student and report measured energy per utterance. Otherwise, energy remains an operation-count estimate and is labeled as such throughout.
References
- Towards end-to-end spoken language understanding
- Low-Activity Supervised Convolutional Spiking Neural Networks Applied to Speech Commands Recognition
- A surrogate gradient spiking baseline for speech command recognition
- Deep Spiking Neural Networks for Large Vocabulary Automatic Speech Recognition
- SpikingBERT: Distilling BERT to Train Spiking Language Models Using Implicit Differentiation
- Event-Driven Language Models with Sparse Neural Activity for Neuromorphic Hardware
- Speech Model Pre-Training for End-to-End Spoken Language Understanding
- Incremental Online Spoken Language Understanding
- SLURP: A Spoken Language Understanding Resource Package
- SpeechBrain: A General-Purpose Speech Toolkit