Spiking Speech Tokenizers That Stay Quiet When Speech Does

Speech language models now hear and speak through a neural audio codec, a small encoder that turns a waveform into a few streams of integer tokens (one token per codebook per frame, at tens of frames per second). The large model reasons over those tokens. For always-on devices such as earbuds, hearing aids and wearables, the part that must run locally is the encoder. It turns audio into tokens that can be shipped to a phone or server. Today's encoders are dense floating-point convolutional networks that do the same amount of work every frame, whether the input is a fricative, a held vowel, or room noise. We propose a spiking codec encoder. It is built from neurons that communicate by sparse binary events, so each event costs one addition, and silent neurons cost nearly nothing. Three design choices make its tokens both cheap and useful to a downstream speech language model. Tokens are read from the neurons' internal voltages rather than from their spike counts. The first codebook is taught phonetic content during training only. Tokens are emitted only when something has changed.

The claim to test is narrow. At matched downstream quality, measured by recognition and synthesis error through frozen evaluators, a spiking tokenizer spends substantially fewer operations than a low-bit conventional encoder that is gated by a voice-activity detector. The savings must come from sparsity inside active speech, not merely from skipping silence. If the savings vanish once silence is gated away, the neuromorphic ingredient is not earning its keep. The experiments are designed to show exactly that if it is true.

What exists and what is missing

Two groups of work motivate this proposal, and neither attempts it.

The first group is spiking audio models, all of which regress continuous targets. DPSNN enhances raw waveforms with a time-domain spiking network at 5 ms latency. It mixes neurons with learnable leak rates for local structure and neurons whose firing threshold rises after each spike for long context. We reuse this encoder design. It paid for time-domain operation with lower perceptual quality, reporting PESQ 2.37 against 2.66 for frequency-domain models. That cost is a constraint we must face. SpikeVoice synthesizes mel-spectrograms with a spiking Transformer at 10.5% of the energy of FastSpeech2, with LJSpeech MOS 4.06 versus 4.10. It shows spiking networks can carry speech-quality information, but it never emits discrete tokens.

The second group is spiking discrete models outside audio. Spiking-Diffusion builds a spiking vector-quantized autoencoder for small images. It needed an extra learned module to recover spike trains from code indices because information was lost between spikes and codes. That finding shapes our decision to quantize membrane voltages rather than spike rates. Sorbet makes a text Transformer hardware-compatible with bit-shift softmax and normalization, reporting 27.16× energy savings over BERT-base. It works on text tokens only.

On the codec side, SoundStream established residual vector quantization. In this scheme, each successive codebook quantizes what the previous ones missed, and VALL-E made those tokens the substrate of zero-shot speech synthesis. X-Codec showed that purely reconstruction-trained codecs give tokens that cause word omissions and hallucinations downstream. Adding self-supervised semantic features halves VALL-E's LibriTTS WER, from 6.37 with EnCodec to 3.26. X-Codec runs a large self-supervised model inside the tokenizer, which is exactly what an always-on device cannot afford.

The research file finds no spiking codec. The gap is real, but "make EnCodec spike" is not by itself an idea worth a paper. What follows is the specific version we think is.

Core Claim

The work of a speech encoder is highly redundant over time. The first codebook changes at roughly phone rate. Deep residual codebooks carry small corrections that are often near zero during stationary segments. A spiking encoder with adaptive thresholds turns this redundancy into skipped work at the level of individual neurons. An event-driven emission rule turns it into skipped tokens at the output. The tokens can be made semantically useful without any semantic model at inference, by distilling self-supervised phonetic targets into the first codebook during training only. This moves X-Codec's semantic cost entirely off-device.

Research Program

The encoder follows DPSNN's time-domain layout: a short-kernel convolutional front end, then stacked spiking convolution and spiking recurrent blocks. It is trained with surrogate gradients, meaning a smooth stand-in for the spike function's derivative is used during backpropagation. There is no STFT buffer, so algorithmic latency stays at one frame hop.

At each frame, residual vector quantization is applied to the membrane voltages of the final layer, not to its spike counts. A voltage is a running sum of weighted spikes, so this needs only additions. Reading it avoids the rate-coding information loss that forced Spiking-Diffusion to add a recovery module. We test this choice directly against quantizing spike rates.

During training, a frozen HuBERT-style model supplies targets. A small head predicts them from the first-codebook embedding, following X-Codec's semantic objective. At inference, the teacher and head are discarded, leaving an encoder that contains only spiking layers and codebook lookups. The decoder is a conventional codec decoder, because in the target use it runs on the phone or server.

The emission rule mirrors a neuron's threshold. Let $r_{k,t}$ be the residual vector entering codebook $k$ at frame $t$. Let $c_{k,t}$ be its chosen code and $\theta_k$ a per-codebook threshold that is learned or scheduled. Codebook $k$ emits a new token only if

$$\lVert r_{k,t} - r_{k,\tau_k} \rVert_2 > \theta_k,$$

where $\tau_k$ is the last frame at which codebook $k$ emitted. Otherwise the receiver repeats the previous code. The thresholds become a bitrate knob. Deep codebooks, whose residuals are small and slow, should fall silent most often. A straight-through estimator lets training see the repeat operation.

Energy is reported two ways. The first is operation counts converted with the per-operation accounting that SpikeVoice and Sorbet use. The second, if hardware access allows, is measured power on a Loihi 2 or Xylo-class device. We will state plainly that simulated counts are the weaker evidence, since the file notes that both DPSNN and Sorbet stopped at simulation or synthesis.

Baselines

The naive reference is a standard floating-point residual-quantization codec encoder of matched parameter count and frame rate, trained on the same data (LibriSpeech/LibriTTS) with the same losses and discriminators. It fixes the quality ceiling and the dense operation count. No reported result was found on Lacuna for EnCodec or DAC on the full Codec-SUPERB suite. We therefore retrain and evaluate it ourselves.

The strongest efficiency competitor is the same encoder quantized to 1-bit weights and 4-bit activations, following the BiT-style binarization that Sorbet compares against. It is additionally gated by a lightweight voice-activity detector that skips frames judged silent. This is what a hardware engineer would build first. It shares the architecture, data, semantic distillation loss and a matched hyperparameter search budget with the spiking model. The spiking model beats it only if, at equal Codec-SUPERB WER and speaker-verification error, it uses fewer energy-weighted operations on voiced frames alone.

The strongest quality competitor is X-Codec, whose reported VALL-E LibriTTS WER of 3.26 is the target for downstream synthesis. We train an identical VALL-E on our tokens and on X-Codec's. For resynthesis, the reference numbers in the file are AFACodec (D) at 7–8 kbps with 3.07% WER, against 2.89% for original audio on Codec-SUPERB.

Three ablations isolate the key ingredients. The first removes semantic distillation. The second replaces voltage readout with spike-rate readout. The third disables event-driven emission, so that every codebook emits every frame.

Why not just…?

Why not just gate a quantized conventional encoder with voice-activity detection? This is the real threat, and we treat it as the main baseline rather than argue it away. Gating captures silence but not the within-speech redundancy of held vowels and near-zero deep residuals. Whether that redundancy is large enough to matter is an empirical question. We settle it by reporting energy separately on voiced frames and on a silence-stripped version of each test set. If the spiking advantage disappears there, we will report that the benefit is gating, not spiking.

Why not just shorten the token sequence by merging adjacent codes? VALL-E R halves the first-codebook rate by pooling and cuts inference time by over 60%. That speeds up the downstream language model, not the on-device encoder, whose per-frame dense work is unchanged. Our emission rule is a content-adaptive version of merging. We compare it against fixed 2× merging at matched average bitrate, to check whether adaptivity buys quality.

Why not just run a small ANN codec plus X-Codec-style semantics on a phone-class chip? Keeping a self-supervised model in the tokenizer is what X-Codec does. It is outside the always-on power envelope this proposal targets, and our training-only distillation is meant to recover most of its WER benefit. The comparison against X-Codec in VALL-E measures how much benefit is lost.

Proposed Experiments

Training uses LibriSpeech and LibriTTS at 16 kHz. Tokenizer evaluation uses Codec-SUPERB's frozen evaluators: Whisper WER, ECAPA-TDNN equal error rate, emotion2vec accuracy, plus PESQ and STOI. STOI matters because Codec-SUPERB reports that it correlates above 0.8 in magnitude with the downstream task metrics. Downstream usefulness is measured by VALL-E trained on each tokenizer, reporting LibriTTS WER and speaker similarity. Streaming robustness is checked on VCTK and on noisy DNS-benchmark speech against DPSNN's PESQ of 2.37 and STOI of 0.93.

We sweep emission thresholds to trace operations-versus-WER curves for every system, including the gated low-bit baseline. The decisive plot is that curve restricted to voiced frames. A second decisive check asks whether semantic distillation closes most of the gap between our tokens and X-Codec's in VALL-E WER. A third reports the fraction of silent neurons and suppressed tokens per codebook depth. If the core claim holds, deep codebooks and stationary segments should dominate that silence.