Where Drafts Break: Syntax-Conditioned Acceptance Models for Speculative Decoding

Speculative decoding speeds up large language models by letting a small, cheap "drafter" guess several upcoming tokens. The large "target" model then checks all the guesses in one parallel pass and keeps the longest prefix it agrees with. Output quality is unaffected, because a rejection-sampling rule guarantees the text comes from exactly the target's distribution. The speedup depends entirely on how long the drafter should keep guessing, and where it should hedge by proposing several alternatives. Current methods decide this from the drafter's own confidence. They also tend to assume that each guessed token is accepted independently of the others, as if every position were a coin flip with the same bias. Language does not behave that way. Our hypothesis is that rejections cluster at predictable structural positions: the opening of a new phrase, the start of a clause, the first word of a new discourse unit. They are rare inside a phrase that is already under way. A drafter can also be confidently wrong at exactly those openings.

We propose to measure this directly. We will then build a cheap acceptance model conditioned on syntactic state and use it to shape draft trees: branch at predicted phrase openings and extend a single chain inside phrases. Verification is left untouched, so decoding stays exactly lossless. The two-sentence version is this. Speculative decoding wastes its draft budget because it treats token acceptance as a uniform, position-blind process, whereas hierarchical syntactic structure predicts where the target model will disagree. If a lightweight syntactic signal forecasts rejections better than drafter confidence alone, spending the draft budget according to it should raise accepted tokens per step at no quality cost.

Core Claim

The claim is empirical and can be falsified in its first stage. Conditioned on the drafter's probability for a token, the chance that the target rejects it still depends substantially on the token's position in the constituency and discourse structure of the text. Examples of that position are depth of open constituents, distance from the last phrase boundary, and whether the token begins an elementary discourse unit (the clause-sized pieces from which rhetorical structure is built). The second part of the claim is that this dependence can be exploited by a draft-shaping rule without running a parser at inference time.

The theoretical opening comes from Fast Inference via Hierarchical Speculative Decoding. That paper finds the optimal chain of drafters by reducing the problem to a shortest-path search. Its latency analysis explicitly assumes independent, identically distributed acceptance events, and the authors name relaxing this as open. Under that assumption, accepted-run lengths are geometric. We replace the constant acceptance rate with a per-position hazard. Let $h_j$ be the probability that the $j$-th drafted token is rejected, given that tokens $1..j-1$ were accepted. The expected number of accepted tokens for a draft of length $K$ is then

$$E[A_K] = \sum_{k=1}^{K} \prod_{j=1}^{k} (1 - h_j),$$

which reduces to the familiar geometric form when every $h_j$ is equal. Our proposal models $h_j$ as a function of drafter confidence plus syntactic state. The resulting estimate of $E[A_K]$ feeds both the stopping rule for linear drafts and the node-expansion rule for draft trees.

Relation to Prior Work

Several speculative methods already adapt draft shape, and each defines the shape from surface statistics. OPT-Tree greedily grows a draft tree to maximize expected acceptance length. It uses cumulative drafter probability as the acceptance estimate and reaches a mean acceptance length of 7.74 on LLaMA-2-70B, compared with 4.97 for EAGLE. It supplies our tree-construction machinery, and we change only its acceptance estimate. Hierarchical Speculative Decoding with Dynamic Window stops drafting when token entropy crosses a threshold, and it stacks 68M, 7B and 70B models. Its modest gains on GSM8K and HumanEval motivate us: they suggest that drafter uncertainty is a poor proxy for disagreement with the target on structured text. Ouroboros drafts in "phrases," but those phrases are n-gram fragments reused from history, not grammatical units. Graph-Structured Speculative Decoding merges draft branches that share identical token strings. Both are complementary to our work: they change what is drafted, whereas we change where the budget is spent.

The "hierarchy" in the two hierarchical speculative decoding papers is a hierarchy of model sizes, not of linguistic structure. We keep that sense of hierarchy intact and add the linguistic one.

On the syntactic side, the idea that uncertainty over grammatical derivations marks natural cut points comes from Grammar Specialization through Entropy Thresholds. That work cut and-or parse trees at high-entropy nodes and obtained roughly 60x faster parsing on ATIS. We borrow its intuition that structural choice points, not individual tokens, are where variance lives. Joint Syntacto-Discourse Parsing and the Syntacto-Discourse Treebank gives a joint constituency-plus-rhetorical parser with 95.4 discourse-segmentation F1. This makes silver phrase and discourse-unit boundaries on generated text reliable enough to serve as training labels. Considering Nested Tree Structure in Sentence Extractive Summarization with Pre-trained Transformer shows that pretrained encoders benefit from explicitly nesting dependency structure inside discourse structure. That result motivates using both levels as features rather than only one.

One line of work might appear to overlap but does not. Grammar-Aligned Decoding makes outputs obey a formal grammar without distorting the model's distribution. It is a sampling method that can require up to 2,000 samples per problem, not an acceleration method. We impose no grammar on the output at all.

Research Program

The first stage is a diagnostic study. It decides whether the project continues. We run standard speculative decoding with an EAGLE-style drafter and a LLaMA-2-70B-class target over the six Spec-Bench tasks described in the speculative decoding survey, which are MT-Bench, WMT14, CNN/DM, Natural Questions, GSM8K and RAG, plus HumanEval. For every drafted token we log drafter probability, drafter entropy, position in the draft, and whether it was accepted. Offline, we parse the final outputs with a constituency and discourse parser, and for HumanEval with an exact Python parser. Each drafted token is then tagged with syntactic features: open-constituent depth, tokens since the last constituent boundary, whether the token opens a constituent, and whether it opens a discourse unit.

We fit three nested hazard models, all logistic regression and then a small multilayer perceptron. The first uses drafter features only. The second adds features from the drafter's hidden state. The third adds the syntactic features. The quantity of interest is held-out log-loss and calibration of rejection prediction, especially the third model's gain over the second. We also measure the empirical departure from the independence assumption: the dispersion of run lengths relative to a geometric fit, and the lag-autocorrelation of acceptance.

The second stage makes the signal cheap. Running a parser during decoding would erase any gain. We therefore train a small classification head on the drafter's hidden states. It predicts the silver syntactic labels (boundary, open depth bucket, discourse-unit onset) together with rejection itself, using parses of drafter-generated text as supervision. For code, an incremental tokenizer-level parser yields exact structure at near-zero cost. This gives a clean contrast between exact and learned syntax.

The third stage plugs the predicted hazards into OPT-Tree's expansion rule. It also plugs them into the drafter-chain optimization of the hierarchical speculative decoding paper in place of a constant rate. Verification stays unchanged, so losslessness holds by construction.

Baselines

The naive reference points are plain autoregressive decoding and vanilla speculative decoding with a fixed linear draft of tuned length. Every speedup is reported against both.

The strongest existing competitor is OPT-Tree, because it already optimizes exactly the objective we target: expected acceptance length under a node budget. It differs from us only in how acceptance is estimated. The cited result to beat is its mean acceptance length of 7.74 on LLaMA-2-70B with a 500-node budget, and speedups of up to 3.2x on MT-Bench and GSM8K (OPT-Tree). We will re-run it rather than rely on those numbers. It will use the same drafter, target, hardware and node budgets (64, 128 and 500), and its stopping threshold will be tuned on the same held-out split and with the same number of trials as our method.

HSDDW's entropy-window stopping is the second structured baseline. Its reported MT-Bench speedup is 2.91x to 3.13x with 70B, 7B and 68M models (Hierarchical Speculative Decoding with Dynamic Window). We run it with the same model stack when we test the model-chain variant.

For the production-style evaluation, the reference is EAGLE3's reported mean speedup of 1.90x on the SPEED-Bench Qualitative Split, for Llama 3.3 70B with draft length 3 (SPEED-Bench).

Two ablations isolate the key ingredient. The first is a learned rejection predictor on drafter hidden states with no syntactic supervision, trained on the same logs with the same capacity and plugged into the same tree rule. The second is a heuristic proxy that treats punctuation and function words as boundaries.

Beating OPT-Tree means a statistically reliable gain in mean acceptance length and in wall-clock speedup at equal node budget, averaged over Spec-Bench. Beating the hidden-state ablation matters more scientifically. It means the syntactic model wins on held-out log-loss and on acceptance length, and above all it wins on SPEED-Bench domains unseen during predictor training.

Why not just…?

Why not just train a rejection predictor on drafter hidden states, with no syntax? This is the real threat. Hidden states may already encode phrase structure, in which case syntactic supervision adds nothing. The second-versus-third model comparison in stage one, and the hidden-state ablation in stage three, settle the question directly. Our bet is not that syntax beats hidden states in-domain. It is that syntactic labels act as an auxiliary target that makes the predictor transfer across domains, because structure varies less across domains than vocabulary does. SPEED-Bench was built to reduce pairwise prompt similarity by 40% relative to SpecBench, so it is the right test. If the ablation matches us out of domain too, we will report that syntax is redundant with learned states and still publish the non-independence measurements.

Why not just recalibrate drafter probabilities, for example with temperature scaling, and keep OPT-Tree? Calibration fixes miscalibration that is uniform across positions. Our hypothesis is that miscalibration depends on position: the drafter is overconfident specifically at constituent openings. We test this by fitting a position-blind calibrator as another baseline and checking whether the gap in the rejection-versus-confidence curve between constituent-opening and phrase-internal tokens survives recalibration.

Why not just use phrase reuse as in Ouroboros? Ouroboros changes what gets drafted and depends on repetition in the context. Its authors note that the phrase pool helps little without that locality. It is compatible with our budget allocation, so we test the combination, not a replacement. If the combination shows no additive gain over Ouroboros alone on low-repetition SPEED-Bench domains, the idea has little practical value there.

Why not just use punctuation as a boundary signal? It is nearly free, and if it captures most of the effect, the elaborate version is unjustified. The heuristic ablation measures this. We expect it to catch clause and sentence onsets but miss phrase openings inside clauses. That is where the stage-one analysis will show whether the extra signal lives.

Decisive Checks

Three results would each end or redirect the project. The first is that in stage one the syntactic features improve held-out rejection log-loss over drafter plus hidden-state features by a negligible margin on every Spec-Bench task. In that case we stop and report the measured non-independence as a characterization result. The second is that the learned syntax head fails to recover most of the gain of oracle parse features. If so, the method is only useful for code, where exact parsing is free, and we would narrow the claim to HumanEval-style generation. The third is that the gain in acceptance length disappears at batch sizes where verification becomes compute-bound. SPEED-Bench documents this regime, and we evaluate across its throughput buckets. That outcome would limit the contribution to latency-oriented serving.

If all three checks pass, the result would be the first draft-shaping rule that uses the hierarchical structure of language. It would also replace the independence assumption in hierarchical drafter optimization with a measured, structure-dependent hazard.