Resonant Spiking Gates for Always-On Addressee Detection

A voice assistant in a shared room has to keep listening, but most of what it hears is people talking to each other. Deciding whether an utterance is meant for the machine is called addressee detection. The most accurate current detectors are large neural networks, from 79-million-parameter Conformers to GPT-2-scale language models, and running them continuously drains the batteries of earbuds, hearing aids and small robots. We propose a small spiking neural network that stays on all the time and decides, within the first second or two of speech, whether to wake the expensive detector at all. A spiking neural network is one whose neurons communicate with sparse binary pulses, so it does almost no arithmetic when nothing relevant happens.

The bet rests on an old finding. People speak to machines with a distinctive rhythmic, "sing-songy" loudness pattern and altered vocal effort. This style alone separates machine-directed from human-directed speech better than transcribed words do, and it does so without running speech recognition (Learning When to Listen; Addressee detection using temporal and spectral dimensions of speaking style). A rhythmic signature is exactly what oscillating spiking neurons are built to pick up. Resonate-and-fire neurons behave like tiny tuned oscillators and fire only when the input matches their preferred rhythm. Our claim is that a bank of such neurons can serve as an always-on addressee gate. It should cut total system energy substantially compared with conventional cheap gates at matched detection error, and the saving should come specifically from their oscillatory, event-driven character.

Core Claim

The energy cost of always-on addressee detection is set mostly by two things: what runs continuously, and how often the heavy model gets invoked. The heavy classifier's own size matters less. Write $E_{\text{gate}}$ for the per-second energy of the always-on stage, $q$ for the fraction of time the heavy detector is woken, and $E_{\text{heavy}}$ for its per-second energy. System energy per second is then

$$E_{\text{sys}} = E_{\text{gate}} + q\,E_{\text{heavy}}.$$

The detection error is fixed by the cascade. We claim that a resonate-and-fire spiking gate achieves a better trade-off between $E_{\text{sys}}$ and cascade error than three cheaper alternatives:

  • a voice-activity gate, which wakes on any speech and so drives $q$ high in multi-party rooms;
  • a quantized conventional network of matched size, which pays clocked multiply cost every frame;
  • a leaky integrate-and-fire spiking gate. Its neurons only smooth their input (first-order dynamics), so they should need more neurons or more spikes to represent rhythm.

The claim is falsifiable piece by piece, and the baselines below are designed so that any one of these alternatives can beat it.

Relation to Prior Work

Two kinds of work motivate the proposal.

The first is the speaking-style literature. It shows the signal exists and arrives early: an "all-prosody" system reaches 12.5% equal error rate (EER, the operating point where false accepts equal false rejects) against 27.0% for a non-oracle word n-gram, and gives good decisions within the first two seconds (Shriberg et al. 2013). Those systems used Gaussian mixture models over hand-built contour features in an offline setting and never asked what it costs to keep them on.

The second is device-directed speech detection at scale. A Multimodal Approach to Device-Directed Speech Detection with Large Language Models fuses audio encoders, recognizer signals and text into GPT-2 with LoRA. Adaptive Knowledge Distillation for Device-Directed Speech Detection compresses a 79M Conformer into a roughly 5M student. We treat both as the expensive second stage, not as competitors to replace. Neither studies when that stage should run.

Proactive Hearing Assistants that Isolate Egocentric Conversations is the closest on-device multi-speaker system. It anchors on the wearer's own speech and runs dense networks on an Orange Pi and Apple silicon. Our gate needs no self-speech anchor, which addresses that paper's stated failure for passive listeners. RARM rejects side-talk into a "Noise" class, but only after encoding every utterance with a 12-layer Transformer over text. We aim to reject most side-talk before any dense computation happens.

The spiking side supplies the method and the constraints. The harmonic resonate-and-fire state space model in SH²RFSSM gives stable second-order spiking dynamics with learnable frequency and damping. It was evaluated only on generic time series, and we reuse its neuron and discretization. Spiking-S4 and the streaming adLIF quality predictor of Nilsson et al. show that audio spiking networks can stream over log-mel frames at high sparsity. They also constrain us. Nilsson et al. lose accuracy as sparsity rises (Pearson correlation falls from 0.810 to 0.782 at 99.3% sparsity), so sparsity is not free. Nearly all of these papers estimate energy from 45 nm operation counts, the 0.9 pJ accumulate versus 4.6 pJ 32-bit float multiply-accumulate figures used by SpikingBERT, rather than measuring it on hardware.

That comparison is against 32-bit float multiply-accumulates. A fair competitor is an 8-bit integer network, and the file gives no per-operation energy for 8-bit multiply-accumulate. We therefore cannot assume the spiking advantage survives that comparison, and the experiments are built to test it directly.

S²M-Former applies spiking networks to auditory attention, but it decodes the listener's brain signals. It does not decide from conversational audio who is being addressed.

Research Program

We would build a streaming gate with three parts:

  1. Encoder. A 1D-convolutional spiking encoder over 40-band log-mel frames at 10 ms.
  2. Resonator bank. Two or three layers of harmonic resonate-and-fire neurons, whose frequencies are initialized across the 2–8 Hz syllabic range and the slower phrase-level range where the energy-contour signature lives.
  3. Readout. A leaky readout that accumulates evidence and emits a wake signal once a threshold is crossed.

Training uses surrogate gradients through time, which replace the spike's zero-or-infinite derivative with a smooth stand-in. Two auxiliary terms shape the gate:

  • a spike-count penalty that tunes the operating point, how many spikes the gate may spend;
  • a distillation term that matches the heavy detector's soft scores, so the gate learns to defer on ambiguous cases rather than decide them.

Decisions are scored at fixed latencies of 0.5, 1 and 2 seconds after speech onset, which makes the early-decision claim measurable.

Two analyses accompany the system. First, we will check whether the learned resonator frequencies and the per-utterance spike counts align with the energy-contour discrete cosine transform and spectral-tilt features of Shriberg et al. 2013. If the gate wins without recovering anything like rhythm, the "resonance" story is wrong even if the engineering works. Second, we will measure how spike count differs between human-directed and machine-directed speech, to test whether side-talk actually stays largely silent in the network.

Baselines

Naive reference. An energy-based voice-activity detector that wakes the heavy detector on all speech. It has near-zero gate cost and the heavy detector's full accuracy, but $q$ equals the speech fraction of the room. Any useful gate must reduce $E_{\text{sys}}$ relative to this at no more than a fixed EER increase, which we set at 1 point absolute.

Classical prosodic gate. The all-prosody Gaussian mixture system of Shriberg et al. 2013, re-implemented with encon, tilt and devo features. It reports 12.5% EER in-domain on the HHC corpus and 22.8% when trained on out-of-domain ATIS plus ICSI data. It is the strongest competitor on the question of whether rhythm can be read cheaply, because its feature extraction is already inexpensive. We count its operations exactly, including feature extraction. Beating it means lower EER at equal operation cost, or equal EER at clearly lower cost, at the 2-second decision point.

Quantized conventional gate. The same architecture with spikes replaced by 8-bit continuous activations, and alternatively a small 8-bit convolutional-recurrent network. Each gets the same parameter budget, training data and hyperparameter search size (an identical number of trials). This baseline isolates whether spiking itself matters.

First-order spiking gate. The same network with leaky integrate-and-fire or adLIF neurons in place of resonators, as in Nilsson et al.. This ablation isolates the oscillatory ingredient.

Heavy stage, run alone. For the expensive second stage we use a distilled roughly 5M Conformer student following Alshawi et al., trained on our public data. That paper reports only relative gains, 26% EER reduction on keyword invocations and 19% on keyword-free follow-ups, on in-house data. The 6.53% EER of Wagner et al. is likewise on an in-house Apple set. Neither number is comparable to ours, so we report the heavy stage's own EER on our test sets as the accuracy ceiling.

Why not just…?

Why not just use a wake word? Wake words fail on keyword-free follow-up turns. These are a primary interaction class in Alshawi et al., and frictionless interaction is the stated goal of A Study for Improving Device-Directed Speech Detection. Spiking keyword spotters already exist (A surrogate gradient spiking baseline for speech command recognition A surrogate gradient spiking baseline for speech command recognition). We include a wake-word-only operating point in the follow-up-turn evaluation to show the recall it loses.

Why not just quantize a tiny conventional network and run it on a microcontroller? This is the real threat. The 5.1x accumulate-versus-multiply figure compares against 32-bit float, not 8-bit integer, and the file reports no 8-bit energy figure. A dense 8-bit gate may therefore close most of the gap. We do not argue this away. The matched quantized-gate baseline, operation counts priced under several published cost assumptions, and physical measurement on Loihi 2 versus a Cortex-M-class microcontroller where hardware access allows will decide it. If the 8-bit gate matches at equal measured energy, the honest conclusion is that resonant dynamics, not spikes, carry the value, and we would report it that way.

Why not just keep the Gaussian mixture prosody classifier? It is cheap and already strong at 12.5% EER. However, it relies on hand-built segment features computed after the fact, and its out-of-domain EER jumps to 22.8%. The resonator gate must beat it on out-of-domain transfer and early-latency EER, or it adds nothing.

Why not just transcribe and classify the text? Lexical models are the weak modality here: 52.62% accuracy from utterance features alone in Jovanovic et al., and a 71% error rate on human-directed speech in Shriberg et al. 2012. Always-on speech recognition is also itself the cost we are trying to avoid.

Proposed Experiments

Data. The primary benchmark is the HHC human-human-computer corpus with its published protocol, if it can be obtained. Because availability is uncertain, we also follow the out-of-domain protocol of Shriberg et al. 2013: train on ATIS-style human-computer and ICSI meeting human-human speech, and test across corpora. Noise augmentation draws on DNS Challenge audio.

For multi-party and passive-listener conditions, we evaluate on the Vernissage robot recordings. These have addressee annotations with kappa 0.93 and servo-motor noise and ego-motion that break vision-based cues. We also use the egocentric binaural corpus of Hu et al., where we drop the self-speech anchor to test the passive case that system cannot handle.

Metrics. We report:

  • gate EER at 0.5, 1 and 2 seconds;
  • cascade EER;
  • wake fraction $q$;
  • $E_{\text{sys}}$, both as priced operation counts and, where possible, as measured on-chip energy.

Decisive checks. The proposal succeeds only if the resonator gate dominates the 8-bit conventional gate and the first-order spiking gate on the curve of $E_{\text{sys}}$ against cascade EER. It fails if either ablation matches it.

As a secondary check, side-talk should produce clearly fewer spikes than machine-directed speech in the gate. If spike counts are similar across classes, the event-driven savings argument collapses, whatever the accuracy.