Shortest-Decision-First: Preemptive Scheduling for Shared Wake-Word Verifiers

Voice devices increasingly share one small neural processor across several audio streams, such as the beams of a microphone array, the satellite microphones of a smart-home hub, or several keyword sets. Each stream runs a tiny always-on detector, and every time it fires, the shared processor runs a heavier second-stage model, the verifier, which confirms or rejects the trigger. Some verifier jobs settle quickly: a clean "Hey Snips" is accepted within a few audio chunks. Others drag on: overlapping TV dialogue in café noise keeps the verifier uncertain until a timeout. When jobs are served first-come, first-served, one of these slow, ambiguous jobs sits at the front of the queue and delays every trigger behind it, including the real wake word. Queueing researchers call this head-of-line blocking. We propose to treat the verifier as a preemptible queue server. A one-bit "will this resolve fast?" hint, computed from the cheap first-stage detector, would order the jobs, and a slow job would be paused at chunk boundaries rather than killed.

What makes this worth doing is a coincidence specific to wake words. In this setting, job size is correlated with the label. True wake words tend to be short jobs, and false triggers tend to be long, ambiguous ones. A scheduler that favors short jobs therefore also favors the jobs whose latency users notice. It also lowers average queueing delay. A second fact makes it cheap. Pausing a large language model request means evicting gigabytes of cached attention state. Pausing a streaming keyword verifier means saving a few kilobytes of convolution or recurrent state. The main cost that limits preemption in LLM serving, the swapping and recomputation that Beyond Prediction: Tail-Aware Scheduling for LLM Inference and ConServe work hard to contain, largely disappears here.

Core Claim

On a shared edge verifier with heavy-tailed time-to-decision, we expect two ingredients together to cut tail response latency for true wake words at matched false-accept and false-reject rates, compared with first-come-first-served, hard timeouts, and prediction-free attained-service scheduling. The first ingredient is a learned one-bit hint about decision time taken from the first-stage detector. The second is cheap chunk-level preemption that pauses slow jobs and resumes them later. The contribution is a measured scheduling result: a specific policy, a model of its preemption cost on edge hardware, and an empirical answer to whether the hint adds anything beyond what attained service already provides. It is not a new acoustic architecture.

Grounding in Prior Work

The queueing side supplies the method. Queues with Small Advice shows in synthetic M/G/1 queues that a single noisy small/large bit, combined with preemption, recovers most of the gap between blind FIFO and shortest-remaining-processing-time. At load 0.95 with Weibull service times, average time in system falls from 58.0 under FIFO to 9.846 with noisy one-bit advice, against 4.517 for the perfect-knowledge optimum. The paper also explains why noisy advice is tolerable. Calling a huge job small causes congestion, while calling a small job large costs little. We test whether this asymmetry holds when the "jobs" are real acoustic verification episodes and the advice comes from a microcontroller-scale model. The paper only analyzes synthetic distributions.

Three LLM-serving papers bound the design. DON'T STOP ME NOW (TRAIL) shows that a probe on a model's internal activations predicts remaining work better than the input alone. Its probe has about 2.1M parameters and reaches 2.66× lower error than a BERT prompt predictor. We borrow the idea of probing internal states. Our probe, however, reads the first-stage detector, predicts a single bit rather than a length, and must cost tens to hundreds of parameters. Beyond Prediction (UniBoost) constrains us. It reports that even oracle shortest-remaining-time scheduling has poor tail latency, and that soft priorities based on attained service beat it by 35–50% on P99 time-to-last-token. We therefore make attained-service scheduling a first-class baseline, and we include an anti-starvation term rather than strict shortest-first. ConServe shows that preemption at layer boundaries ("safepoints", about 13 ms, about 1.1% overhead) removes blocking between online and offline jobs on GPUs. We place safepoints at audio-chunk boundaries of a streaming verifier. There the saved state is tiny and resuming requires no recomputation.

The keyword-spotting side supplies the components and the motivation. MFA-KWS and TDT-KWS skip redundant frames, reporting 47–63% and up to 4× speedups respectively. Skipping makes per-job compute depend on content, which means attained compute is no longer a clean proxy for remaining work. That observation motivates measuring attained service in audio consumed rather than compute spent. Latency Control for Keyword Spotting shows that standard max-pooling loss trains "lazy" detectors that wait for post-keyword context. Its stochastic nudge yields a 25% relative false-accept reduction at fixed latency. This is a competing way to shorten jobs, and we treat it as complementary, since it acts on the verifier's training rather than on the queue. The small first-stage models come from MarbleNet (88K parameters) and compact detectors like the 12K-parameter SWSA-TDNN of Jiangyan Yi et al.. No paper in the file schedules a shared wake-word verifier as a queue.

Research Program

We first build an anytime streaming verifier. This is a model that consumes audio chunk by chunk, keeps a running confidence, and stops as soon as it is confident to accept or reject, up to a maximum horizon. Its time-to-decision is the job size. We start from the CTC branch of an MFA-KWS-style model and from a cFSMN-style streaming model (Mengzhe Chen et al.), with sequential accept/reject thresholds. The first deliverable is purely measurement. We will record the distribution of time-to-decision on positives and on hard negatives, and how strongly it correlates with the label. If the distribution is not heavy-tailed, or size does not separate true triggers from false ones, the premise fails and we stop. This check is cheap and comes first.

Second, we build the advice signal. A tiny logistic probe reads the first-stage detector's penultimate features and score at the moment it fires. It is trained on verifier logs to predict whether the verifier will decide within $k$ chunks. We compare it against using the raw first-stage score as the bit, to see whether learning adds anything.

Third, we build the scheduler and its preemption mechanism. Each job's state is checkpointed at chunk boundaries, and its unprocessed audio sits in a per-stream ring buffer. The scheduler ranks runnable jobs first by the advice bit and then by least attained audio, with an aging term so long jobs eventually run. Jobs that catch up to live audio give up the processor until new audio arrives. We measure the per-job state size and the context-switch time on a Cortex-M-class microcontroller and on a phone digital signal processor. These measurements feed a calibrated simulator, which lets us sweep load and the number of streams.

Baselines

The naive reference point is first-come-first-served without preemption. Each trigger runs the anytime verifier to completion in arrival order. This is what the cascaded pipelines implied by MFA-KWS and KWT do when treated as atomic verification calls.

The strongest existing acoustic method is the same queue with an MFA-KWS-style frame-skipping verifier. Its 47–63% speedup over frame-synchronous decoding shortens every job and so lowers load. It runs on the same audio, the same processor model, and the same accept thresholds, tuned to the same false-accept-per-hour target. The file reports no wake-word queueing latency figure for MFA-KWS or any other KWS system, so we report none.

The strongest existing scheduling method is prediction-free attained-service priority in the style of UniBoost. That paper reports 35–50% lower P99 time-to-last-token than oracle-length SRPT. We port its idea directly, ranking jobs by least audio processed with the same aging term, and give it the same tuning budget as our policy: a grid over the aging parameter, selected on a held-out stream set.

We also run an oracle that knows each job's true time-to-decision. It gives the ceiling, analogous to the 4.517 shortest-remaining-time figure in Queues with Small Advice.

Two ablations isolate the key ingredients: the advice bit with no preemption, and preemption with no advice, which is simply the attained-service baseline. Our policy beats the field if it lowers P99 true-wake-word response latency below attained-service scheduling by a margin larger than the across-seed spread, at false-accept and false-reject rates matched within confidence intervals. Beating first-come-first-served alone would not count as success.

Why not just…?

Why not just cap verifier time with a hard timeout? A timeout bounds job size and so bounds blocking, and it is the most serious threat. Its cost is that it forces a reject on hard true wake words, such as far-field or noisy ones, raising false rejects exactly where users already struggle. Preemption deprioritizes those jobs instead of discarding them. We will sweep the timeout and plot P99 true-trigger latency against false-reject rate for both approaches. If some timeout matches our curve, the honest conclusion is that a timeout suffices, and we will report that.

Why not just use attained-service or round-robin scheduling and skip the advice? For heavy-tailed sizes with decreasing hazard rates, least-attained-service scheduling is already near-optimal without any prediction, and UniBoost's results favor it. What it cannot do is tell apart two newly arrived jobs, one a clean wake word and one TV dialogue, until both have consumed audio. During simultaneous bursts across beams, that early window is where true-trigger latency is set. The advice-with-preemption arm against the preemption-only arm settles this directly. If they tie, we learn that the hint is unnecessary.

Why not just train the verifier to decide earlier? The latency nudge of Latency Control for Keyword Spotting shortens decisions on keywords. We expect it to leave ambiguous negatives long, because the verifier is uncertain about them, not lazy. We will train verifiers at several nudge values and run every scheduler on each. The prediction is that scheduling gains persist after nudging. If nudging collapses the tail on its own, the scheduler is unnecessary.

Why not just make the first stage accurate enough that few false triggers reach the verifier? Fewer false triggers means lower load and less blocking. Tiny models, however, have a capacity ceiling in noise, which MFA-KWS documents with WHAM!-mixed audio. We will sweep first-stage size from 12K to 88K parameters and report where queueing stops mattering.

Proposed Experiments

Positives come from Hey-Snips and MobvoiHotwords, with LibriKWS-20 for custom keywords. Negatives and background mix in WHAM! noise and long non-keyword speech. We build multi-stream workloads by assigning independent sessions to 2–8 simulated beam or room streams. We also inject correlated bursts, where the same TV segment reaches several streams at once, to mimic one loud source heard everywhere.

Metrics are the median and P99 time from keyword end to accept decision for true wake words, false accepts per hour, false rejects, ring-buffer overflow rate, and processor energy per hour of audio.

Three checks decide the outcome. The first is the premise test on time-to-decision heavy-tailedness and label correlation. The second is the head-to-head against timeouts and attained-service scheduling at matched false-accept and false-reject rates across load levels. The third is an advice-noise sweep, injecting errors into the bit, to test whether the asymmetric-error tolerance seen in Queues with Small Advice carries over to acoustic jobs.