A voice assistant that turns speech into an executable command, such as "set an alarm for seven unless it's a holiday," faces two pressures. It should answer quickly, ideally with the command ready the moment the speaker stops. It should also be cheap, so a large model should only be called when a small one is likely to get the command wrong. The obvious design has three steps. First, wait until the user stops talking. Second, have a small model produce the command. Third, have a router decide whether the whole command goes to a large model. This design pays the full silence-detection delay, which dialogue systems have long set at 750–1500 ms (TRIPS). It also escalates everything or nothing.
We propose a different design. A small model writes the command as a partial structured parse while the user is still speaking. Grammar-constrained decoding keeps that parse valid at every step. This technique masks out any next token that would break a formal grammar, so the partial output is always a well-formed prefix of a legal command. Because the partial output has structure, the system can decide for each open slot of the command whether to commit it, keep listening, or hand only that sub-part to a large model. The large model then continues decoding from the small model's grammar-valid prefix instead of starting over. The claim is that the parser state gives a better signal than utterance-level confidence for deciding when and what to escalate, and that this signal stays useful even though streaming transcripts keep getting revised.
Core Claim
The unit that should be escalated in a streaming spoken-command system is the open nonterminal of a grammar-constrained partial parse, not the utterance. A nonterminal is a slot or sub-command in the grammar, such as an intent label, a date argument, or a nested sub-intent. Its deferral decision should depend on three things. The first is the grammar category the slot belongs to. The second is whether the small model is filling the slot with a value or leaving it empty. The third is how stable the underlying speech recognition (ASR) words have been across recent revisions. With this, a cascade can send a minority of slots, rather than utterances, to the large model mid-utterance. The target is to match the accuracy of a large-model-only system while cutting the delay between end of speech and a ready command well below that of an endpoint-then-route cascade.
Three findings motivate the design. Improving Cascade Routing for Structured Attribute Generation with Heterogeneous Confidence shows that a single global confidence threshold is the wrong tool for structured outputs. That paper conditions deferral on whether a small Qwen3-4B predicts a value or abstains, and on the attribute family. It reaches 0.8146 precision while routing 19.3% of traffic to Qwen3-235B, compared with 26.4% under a global threshold. Its setting is static catalog text. We move the same idea to parser states that change over time. Can Speech LLMs Think while Listening? shows that information arrives unevenly across an utterance and that a fixed trigger point fails. That paper replaces the fixed trigger with a KL-divergence "question completeness" score, but it names its single global threshold as a limitation. Stream RAG (Stream RAG) shows that issuing tool calls mid-utterance can hide latency, reaching −0.68 s first-token latency on SLUE-SQA. It generates those calls without constraints and reports garbled structured tokens, for example "one1995" for "The 1975".
Research Program
The system has four parts.
The first part is streaming recognition. A streaming recognizer emits a revised hypothesis at each audio chunk. The HuBERT-based recognizer benchmarked on STOP (STOP) reports 4.26% word error rate there and is a reasonable starting point.
The second part is the small parser. A small language model decodes the STOP compositional intent/slot bracket format under a context-free grammar of intent and slot labels. We add an input-dependent grammar in the style of Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning. Under this grammar, slot values must be spans copied from the current transcript. In that paper, input-dependent grammars raised entity disambiguation accuracy from 54.1% to 80.3% at 1–4 ms per token. In our setting the copy constraint should also stop phonetic garbling of entities of the kind Stream RAG reports. We handle subword-to-terminal alignment with the transducer construction of Flexible and Efficient Grammar-Constrained Decoding.
When the transcript is revised, the parser rolls back to the longest parse prefix whose copied spans still match the new transcript. This rule handles speech repairs such as "Boston—no, Denver" without re-parsing from scratch, in the spirit of Detecting and Correcting Speech Repairs.
The third part is the per-slot decision. At each chunk, every open or recently closed slot $s$ gets an estimated probability $\hat p_s$ that its current filler will equal the final gold filler. A small calibrator predicts $\hat p_s$ from:
- the slot's grammar category $g(s)$,
- its prediction type (value or empty),
- the small model's token log-probabilities over the slot,
- the number of recent ASR revisions that touched the slot's span.
The rule escalates slot $s$ when
$$\hat p_s < \tau_{g(s)},$$
where each threshold $\tau_{g}$ is fit per grammar category to meet a target escalation budget. The rule commits a slot when $\hat p_s$ is high and its span has been stable for the last $k$ chunks. Otherwise it waits. Committing matters because committed slots can start downstream work, such as prefetching or tool calls, in the way Stream RAG does.
The fourth part is the handoff to the large model. Escalation passes the transcript and the grammar-valid parse prefix, written as grammar terminals rather than tokens, to the large model. The large model resumes constrained decoding at the escalated nonterminal under the same grammar. Because the prefix is in terminal space, the small and large models can use different tokenizers. This removes the shared-tokenizer requirement of Constrained Decoding with Speculative Lookaheads. That paper verifies a small model's drafts with a large model against a scalar reward on fixed text prompts. We do not use verification of every draft. We use selective, structure-scoped handoff over changing input.
The analysis has two parts. The first asks whether grammar category and transcript stability actually explain variance in slot correctness beyond log-probabilities. The second measures how premature commitment trades off against latency.
Baselines
The naive reference points are the two single-model systems. Both run on the same streaming transcripts and wait for the endpoint.
- Small model alone gives the cost floor.
- Large model alone gives the accuracy ceiling. It uses the same grammar-constrained decoding and is run once at the endpoint.
STOP reports roughly 68% exact match for end-to-end models and roughly 85% for gold-text NLU systems. These numbers bracket what a transcript-based parser can achieve.
The strongest competitor is an endpoint-then-cascade pipeline. The small model decodes the full parse after the endpoint, again under the grammar. The type-aware selective deferral of Improving Cascade Routing for Structured Attribute Generation with Heterogeneous Confidence then decides whether to re-decode the whole utterance with the large model. We adapt it with categories from STOP's slot types. This baseline has the grammar, the calibrated type-aware routing, and the same models. It lacks only streaming and slot scoping, so it isolates what our proposal adds.
That paper's reported number is 0.8146 precision at 19.3% escalation on the e-commerce catalog. No reported result was found on Lacuna for a cascade on STOP. We therefore run the baseline ourselves under the same thresholds-fitting budget on the same development split.
A second strong competitor is a RouteLLM-style learned router (RouteLLM) applied to the final transcript. RouteLLM reports a 50% performance gain on MT-Bench while calling the strong model about 13% of the time. That figure comes from open-ended chat, not structured parsing.
Beating the endpoint-then-cascade baseline means reaching equal exact match at equal or lower large-model token cost, with lower median delay from end of speech to ready command. Alternatively, it means higher exact match at matched cost and latency.
Three ablations isolate the key ingredients:
- Global confidence threshold: replaces the per-category, stability-aware calibrator with a single threshold on mean slot log-probability.
- Utterance-scoped escalation: escalates the entire utterance whenever any slot fails, instead of only the failing slot.
- No grammar: removes grammar constraints from the streaming small model. Parses are validated only after decoding, and invalid ones are escalated.
Why not just…?
Why not just wait for the endpoint and use a good text cascade? This is the most serious alternative. If the large model decodes a short parse quickly, the latency advantage of streaming might shrink to a few hundred milliseconds. The endpoint-then-cascade baseline tests this directly. We will report delay from end of speech to ready command separately from the silence-detection delay. The claim is weakened if its advantage comes only from early endpointing, which any system could adopt. To rule that out, both systems use the same endpointer. Stream RAG's drop from 3.16 s to 1.36 s suggests that mid-utterance work matters. Its setting, however, involves retrieval, not parsing. If our gains fall below roughly the cost of one large-model decode, we will report that slot-level streaming is not worth its complexity for short commands, and limit the claim to long compositional utterances.
Why not just run the large model incrementally on every partial transcript? This removes routing error entirely. It multiplies large-model cost by the number of transcript revisions, the pattern Stream RAG's fixed-interval mode pays for with parallel query threads. We include it as a latency and accuracy upper reference and report its cost. The proposal succeeds only if it gets close to this reference's accuracy at a fraction of its cost.
Why not just use one speech model that thinks while listening? Monolithic systems such as the Moshi extension in Can Speech LLMs Think while Listening? and FLAIR (The Silent Thought) avoid a handoff altogether. Their capacity is fixed by one 7B model, and their outputs are free-form or latent. FLAIR reaches 56.2% on MMSU, a general spoken-understanding benchmark rather than a parsing one. Neither emits a grammar-checked command. We will evaluate the small model alone as a proxy for a capped monolithic system. If it matches the cascade on STOP, cascading is unnecessary at this task's difficulty, and we would say so.
Why not just draft without constraints and validate afterwards? Post-hoc validation catches invalid syntax. It cannot catch a well-formed parse with a wrong slot, and it forces whole-utterance fallback whenever syntax breaks. Constrained Decoding with Speculative Lookaheads reports that unconstrained greedy drafting loses 10–20% on constraint metrics. Geng et al. report that unconstrained LLaMA produces valid parse trees only about 64% of the time. The no-grammar ablation measures what this costs in escalation rate on spoken input.
Proposed Experiments
The main evaluation uses STOP, with audio replayed in real time through the streaming recognizer, because it has compositional nested parses. SLURP (SLURP) serves as a second domain. For reference, its noisy-transcript pipeline reports 81.42% action accuracy and 70.84 SLU-F1, compared with 86.99% and 78.19 on gold text. To stress transcript revision and repairs, we concatenate ATIS utterances into multi-utterance streams, following Incremental Online Spoken Language Understanding, and inject synthetic repairs.
We use two model pairings, a Qwen3-4B / Qwen3-235B pair and a small/large pair with mismatched tokenizers, to test the terminal-space handoff.
The metrics are:
- Accuracy: exact match and slot F1.
- Cost: large-model tokens decoded per utterance.
- Latency: median and 95th-percentile delay from end of speech to a ready command.
- Premature commitment rate: the fraction of committed slots that differ from the final gold filler.
We sweep the escalation budget and plot accuracy against cost and accuracy against latency for every system.
Three checks decide the question.
- Signal quality: stability-aware, per-category calibration must beat the global-threshold ablation in slot-correctness AUROC at every chunk position, not just at the endpoint.
- Slot scoping: slot-scoped escalation must beat utterance-scoped escalation in large-model cost at matched exact match.
- Latency over the pipeline: the full system must strictly improve on endpoint-then-cascade in latency at matched accuracy and cost, with premature commitment below a pre-registered 2%.
If the first check fails, the grammar adds validity but not a better routing signal. The contribution then shrinks to "grammar-constrained streaming parsing," and we would report it as such.
References
- Two Diverse Systems Built using Generic Components for Spoken Dialogue (Recent Progress on TRIPS)
- Improving Cascade Routing for Structured Attribute Generation with Heterogeneous Confidence
- Can Speech LLMs Think while Listening?
- Stream RAG: Instant and Accurate Spoken Dialogue Systems with Streaming Tool Usage
- Stop: A Dataset for Spoken Task Oriented Semantic Parsing
- Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning
- Flexible and Efficient Grammar-Constrained Decoding
- Deyecting and Correcting Speech Repairs
- Constrained Decoding with Speculative Lookaheads
- RouteLLM: Learning to Route LLMs from Preference Data
- The Silent Thought: Modeling Internal Cognition in Full-Duplex Spoken Dialogue Models via Latent Reasoning
- SLURP: A Spoken Language Understanding Resource Package
- Incremental Online Spoken Language Understanding