Repair-Shaped Draft Trees for Fast Verbatim Recognition of Disfluent Speech

Large language models are now used as the text decoder in speech recognizers, and they are slow because they write the transcript one token at a time. Speculative decoding speeds this up without changing the output. A small "draft" model guesses several tokens ahead, arranged as a tree of alternatives, and the large model checks the whole tree in one parallel pass, keeping the longest correct prefix. On read audiobook speech this works very well. We expect it to break down on conversational speech, where people repeat words, restart phrases, and insert "uh". At these points the draft model's guesses are poor, so few tokens are accepted and the speedup disappears. Yet disfluencies are among the most predictable events in speech if you know their shape: after an interruption, a speaker usually repeats the last word or two, restarts from the beginning of the phrase, or says a filler.

This note proposes building part of the draft tree from that shape rather than from the draft model's guesses alone. When a light acoustic detector signals a likely interruption, the tree gains branches that copy recent transcript words (repetition), copy from the last phrase start (restart), or insert a filler. The draft model then extends each of these branches. The large model still verifies every token exactly, so the transcript is identical to plain decoding. The goal is to recover speculative speedup on spontaneous and children's speech, which is where verbatim transcripts matter most for assessment and clinical use.

Core Claim

Speculative acceptance collapses in the tokens around disfluencies. Branches generated from a small grammar of speech repairs, gated and weighted by an acoustic interruption detector, recover most of that lost acceptance at negligible cost. They do this better than what a learned draft model's top-k guesses, draft-token recycling, or a draft fine-tuned on disfluent text can achieve alone. The method is lossless by construction, so the entire claim is about acceptance length and wall-clock speed at fixed word error rate.

Where This Sits

The mechanism comes from dynamic draft trees. EAGLE-2 expands the tree along paths whose product of draft confidences is highest. OPT-Tree chooses the tree that maximizes expected acceptance length. Both rely on the finding in GliDe with a CaPE that draft probability tracks acceptance. We reuse OPT-Tree's expected-length objective. We also add a failure these methods do not treat: near disfluencies, the draft's probability is low and its alternatives are wrong, so widening the tree around its own guesses does not help.

The closest existing work is SpecASR. It brings adaptive truncation, sparse trees at uncertain points, and recycling of rejected draft tokens to LLM-based recognition, reaching 3.04x–3.79x over autoregressive decoding on LibriSpeech with no WER loss. It tests only read speech, and its authors report that speedups degrade when the draft becomes less accurate on noisy input. Its recycling reuses the draft's own rejected tokens. We copy the transcript's already-accepted tokens, because the speaker is repeating themselves, and we place these copies using acoustic evidence of an interruption. That difference is what we test.

Two other lines of work motivate the choice of domain. Streaming Joint Speech Recognition and Disfluency Detection shows that a disfluency label can be predicted jointly with each token from acoustics, which justifies our cheap detector. It never uses that signal to shape the search. The children's reading work in Improving ASR for Children's Reading Assessment with Disfluency-aware Language Models adds repetition and truncation paths to static lattices. It reports that adding them to the graph inflates the search, which forced it to drop harder error types. We add such paths only transiently, where the audio suggests them, and let parallel verification discard them.

From parsing of ungrammatical input we borrow one idea. Eliminative Parsing with Graded Constraints replaces hard rules with weights that penalize rather than forbid. Weighted Interaction of Syntax and Semantics keeps competing readings alive until evidence resolves them. Our repair templates are exactly such soft, deferred hypotheses. We do not claim a parsing contribution.

We also deliberately keep verification exact. Principled Coarse-Grained Acceptance relaxes acceptance to groups of acoustically interchangeable tokens for speech synthesis. For recognition transcripts, relaxing acceptance would change the output. It would also blur exactly the verbatim distinctions, such as a repeated word or "uh" versus "um", that assessment needs.

Research Program

The target system is an LLM-based recognizer in the style of SpecASR: a speech encoder feeding a 7B-parameter decoder, with TinyLlama as the audio-conditioned draft. The decoder is fine-tuned to produce verbatim Switchboard transcripts, including fillers, repetitions, and partial words. This choice matters. Speculation reproduces whatever the large model would output, so if the large model smooths disfluencies away, as the reading-assessment study found Whisper-v3 does, no draft tree can restore them. Verbatim fidelity is therefore handled by the target model's training. Our method is only responsible for speed.

The first step is diagnostic. We run SpecASR-style decoding on Switchboard verbatim transcripts and measure accepted tokens per verification step. We bin the steps by distance to the nearest annotated interruption point. The hypothesis is that acceptance drops sharply within a few tokens of interruptions, and that this drop accounts for most of the gap to LibriSpeech speedups. If the drop is small, the premise fails and we stop here.

The second step builds the detector. It is a small classification head on the speech encoder's frames that outputs, for each point in the audio aligned to the current decoding position, the probability $q$ that an interruption occurs there. It is trained on Switchboard disfluency annotations, following the joint-labeling setup of Streaming Joint ASR and Disfluency Detection. Its cost is one linear layer per frame.

The third step defines the repair templates and scores them. At each draft round, besides the draft's own top-k children, the tree may add three kinds of template branches. A repetition branch copies the last one to three accepted words. A restart branch copies tokens from the most recent phrase start, located with a cheap boundary heuristic on the accepted prefix. A filler branch inserts the speaker's most frequent fillers. The draft model then continues each template branch for a few tokens. We give each node $v$ an estimated acceptance probability $a(v)$ and allocate a fixed node budget $N$ to maximize expected accepted length, as OPT-Tree does:

$$\max_{T,\ |T|\le N} \sum_{v\in T} \prod_{u \in \mathrm{path}(v)} a(u),$$

Here $T$ is the draft tree and $\mathrm{path}(v)$ is the set of nodes from the root to $v$. For ordinary nodes, $a(u)$ is the calibrated draft probability. For the first node of a template branch, $a(u) = q \cdot w_\tau$, where $w_\tau$ is a per-template weight in $[0,1]$ fitted on held-out data, in the spirit of graded constraints. Tokens copied inside a template receive a high fitted continuation probability.

The fourth step is the Spanish children's reading setting from the reading-assessment study. There the passage being read is known, so templates can also propose skipping ahead in the passage or rereading it. This is a second, stronger test, but it depends on obtaining the Reading Fluency Census data and building a Spanish LLM recognizer. We treat it as conditional.

Baselines

The naive reference is plain autoregressive decoding with the same verbatim target model. It fixes the WER that every speculative variant must reproduce exactly, and it defines speedup 1.0x. Its Switchboard numbers will be measured by us. For context on accuracy and latency, the streaming joint model reports 21.3% disfluency-removed WER at 529ms median latency on Switchboard (Streaming Joint ASR and Disfluency Detection). That is a different task framing, so we cite it only as orientation, not as a speed competitor.

The strongest competitor is SpecASR in full, combining adaptive truncation, draft recycling, and two-pass sparse trees (SpecASR). It reports 3.04x–3.79x over autoregressive decoding and 1.25x–1.84x over earlier speculative methods on LibriSpeech, with zero WER loss. We found no reported result on Lacuna for speculative decoding on Switchboard or any disfluent corpus, so we will reproduce SpecASR ourselves. It is strongest because it already branches at uncertain points and already recycles off-track tokens, which covers some of what our templates might capture. We run it with the same target, the same draft, the same node budget $N$, and the same hardware. Its thresholds are tuned with the same number of held-out trials we use to fit $w_\tau$.

We also include an EAGLE-2-style dynamic tree driven purely by global path confidence (EAGLE-2). This tests whether a generic dynamic tree already solves the problem. It reports 3.05x–4.26x on text chat benchmarks, not speech.

Two ablations isolate the key ingredient. The first adds template branches at every step with a constant $q$, testing whether the acoustic detector matters. The second uses the detector to widen the tree with ordinary draft top-k children instead of templates, testing whether the repair content matters. Beating SpecASR means higher mean acceptance length and wall-clock speedup on Switchboard at identical WER. The gain should be concentrated in the near-interruption bins, and speed on LibriSpeech should be no worse.

Why not just…?

Why not just run SpecASR or EAGLE-2 on conversational speech? Their branching expands around the draft's own alternatives. At a repetition, those alternatives are fluent continuations, not the copy the speaker actually produces. SpecASR's own authors report degradation when draft accuracy falls. This is a hypothesis, not a result, and the diagnostic step plus the head-to-head comparison settle it. If SpecASR's acceptance near interruptions is already close to its acceptance elsewhere, our proposal has no room to help.

Why not just fine-tune the draft on disfluent transcripts, for example with synthetic disfluencies like disfluentES? This is the most serious threat. The reading-assessment study shows synthetic disfluent text strongly improves small language models. A draft trained this way may learn to put "I I" in its top-k by itself. We therefore apply the same disfluent fine-tuning to the draft in every speculative arm, including ours. We report whether the templates still add acceptance on top. If they add nothing over a fine-tuned draft, the honest conclusion is that draft data, not tree shape, is the fix, and we would report that.

Why not just copy recent context with no disfluency model at all, as simple prompt-lookup schemes do? Always-on copy branches cost tree budget on the roughly nine in ten steps where no repair occurs. The constant-$q$ ablation measures exactly this. If it matches the gated version, the detector is unnecessary. The claim then narrows to repair-shaped branches alone, which remains a cheap, useful result.

Why not just use classical confidence-guided beam pruning? Confidence-guided dynamic pruning reaches large speedups on Verbmobil, but it trades accuracy for speed, with roughly a 1% WER increase at 14x. It also targets frame-synchronous search over lexical trees. Our bottleneck is token-by-token generation in a large decoder, and we require identical output, so the two are not substitutes.

Proposed Experiments

The primary benchmark is Switchboard with verbatim references. We report mean acceptance length, wall-clock speedup over autoregressive decoding at batch size one, and WER, which must match autoregressive decoding exactly as a correctness check. Every speedup and acceptance number is also stratified by distance to annotated interruption points. LibriSpeech test-clean and test-other serve as a control, where the method should neither help nor hurt. CSJ, used by the joint disfluency paper, tests whether the templates transfer to Japanese repair patterns. The children's reading corpus is the conditional stretch setting.

The decisive checks are four. First, the diagnostic must show a real acceptance drop near interruptions. Second, the full method must close a substantial fraction of that drop relative to SpecASR at equal budget. Third, the gain must survive a draft fine-tuned on disfluent text. Fourth, the two ablations must show which of detector gating and repair content carries the effect.

We also report detector overhead and tree-construction time separately. Construction overhead is the known weak point of adaptive trees (OPT-Tree), so a gain in acceptance length that is eaten by construction time counts as a failure.