Fast language model inference increasingly relies on drafting several future tokens at once and letting the large model check them. This is speculative decoding: a cheap "drafter" proposes a block of tokens, and the target model accepts the longest correct prefix in one pass. Drafting in parallel is fast, but parallel heads predict each slot as if the other slots did not exist. This is the conditional independence assumption, and it produces mixtures such as "San York" from "San Diego" and "New York". Current fixes add sequential conditioning back in. The open question is where to put it. Gumiho hand-picks two serial positions followed by five independent heads, and AngelSpec trains a separate drafter for each workload. Neither split is derived from anything measurable.
We propose to measure, with a small and cheap autoregressive model, how much each future draft slot depends on the slots before it. We call this a dependency profile. We then build the drafter's wiring, meaning which slots get serial conditioning and on which predecessors, directly from that profile under a latency budget. The bet is architectural transferability: the profile is mostly a property of the data domain and tokenizer, not of model size. If so, a profile measured on a 1B model should pick the right drafter wiring for an 8B or 70B target, with no architecture search on the large model.
Core Claim
On the Learning of Non-Autoregressive Transformers shows that the error of parallel generation is bounded below by the conditional total correlation among target tokens, which is the information the tokens share given the context. Total correlation is a single scalar, but it decomposes into per-slot terms, and those terms are what an architect needs. For draft slot $j$ and a candidate set $S$ of earlier slots it may condition on, we define the gain
$$ g_j(S) = \mathbb{E}_{c}\big[\, I(x_j ; x_S \mid c) \,\big], $$
where $c$ is the prompt plus the accepted prefix, $x_j$ is the token at draft slot $j$, $x_S$ are the tokens at the slots in $S$, and $I$ is mutual information.
We compute $g_j(S)$ under a small proxy autoregressive model $q$. The conditional term $\log q(x_j \mid c, x_S)$ is computed exactly when $S$ is the full prefix of slots. Any marginal over excluded slots is estimated by Monte Carlo sampling from $q$.
Under the first-failure rule that Gumiho proves governs acceptance length, an error at slot $j$ discards every later slot. We therefore weight each slot's gain by the probability that verification reaches slot $j$, which gives a weighted profile $w_j \, g_j(S)$. The wiring we build is a small directed acyclic graph over the $K$ draft slots. A slot is conditioned on a set $S$ only if the weighted gain $w_j\,g_j(S)$ exceeds that edge's latency cost, measured as wall-clock time on the target hardware.
What would be new has three parts. First, the serial-versus-parallel split becomes an output of a measurement rather than a design choice. Second, the measurement is made offline on a proxy model rather than at every decoding step. Third, the central empirical claim is that the measurement transfers across scale, and can be re-taken cheaply for a new domain.
Research Program
The work has three stages.
Stage 1: build the profiling tool and test whether profiles transfer. - We profile each domain with Llama-family proxies at 1B and 3B. The domains are math word problems, code, and open-ended chat. - We repeat the profiling with the 8B and 70B target models themselves, so we can test transfer directly. Transfer means the rank correlation of $w_j g_j(S)$ across all slot/predecessor pairs remains high between scales. - This mirrors the cross-scale consistency PLoP reports for its forward-pass feature-norm signal, which picks adapter placement and is consistent from 1B to 3B. PLoP's signal measures how well a module's features align with the data, not dependence between future tokens, so we borrow its methodology but not its metric.
Stage 2: define the space of drafter wirings. - Drafters are built from the components Gumiho and DFly already validated: lightweight serial Transformer blocks and predecessor-conditioned correction heads where conditioning is wired, and independent MLP heads elsewhere. - Following On Network Design Spaces for Visual Recognition, we study a distribution of wirings rather than a single winner. We sample about 30 wirings per target model, spanning fully parallel, Gumiho's 2+5 split, front-loaded and back-loaded serial capacity, and non-contiguous conditioning. - Every candidate is trained at matched parameter count and data. The profile-derived wiring is one point in this population. - Because Decoding Speculative Decoding shows drafter latency is dominated by depth, the latency cost in the selection rule is measured empirically, per added serial layer, on the actual hardware.
Stage 3: test transfer under domain shift. - AngelSpec argues that chat and code/math need different drafters. Our version of that claim is narrower and testable: a domain changes the profile, a re-profiled proxy can detect the change, and the detected change predicts which wiring wins on the large target.
We deliberately stay with subword vocabularies and simple serial heads rather than probabilistic circuits. Fast and Expressive Multi-Byte Prediction with Probabilistic Circuits reports that its tree-structured circuits do not scale past byte vocabularies of roughly 300 symbols, which constrains our design away from them.
Baselines
Naive reference: plain decoding. The do-nothing reference is ordinary autoregressive decoding with the target model. All speedups are reported against it on identical prompts and hardware.
Strongest existing method: Gumiho. Gumiho uses a 2-layer serial Transformer for the first two slots and five parallel MLP heads for the rest, plus Full Tree Attention. It is the strongest competitor because it already exploits the position-dependence of acceptance, just with a hand-set split. It reports a 3.45x speedup on LLaMA-3 70B versus 2.98x for Eagle-2, and 4.5% to 15.8% over Eagle-2 across benchmarks. We will reproduce Gumiho with the same training data, the same number of training steps, the same total drafter parameter count, and the same tree-verification budget as our profile-derived drafter.
Earlier strong method: Eagle-2. Eagle-2 is included as the method Gumiho measured against. We have no Eagle-2 number of our own from the file beyond Gumiho's comparison.
Wiring ablations. Three ablations isolate the key ingredient, which is the profile:
- A random wiring with the same serial-capacity budget as the profile-derived wiring.
- A fully parallel, Medusa-style set of independent heads, matched in parameters.
- A profile computed from the target model's own self-attention between draft positions, in the spirit of DAPD. This tests whether measuring information gain beats using attention as a stand-in for dependence.
Oracle. The best wiring found by exhaustively training all sampled candidates on the target model is an upper bound rather than a competitor.
Beating Gumiho means a higher mean accepted length and a higher wall-clock speedup at equal drafter parameters and training compute, on at least math and code. It also means landing within a small margin of the oracle while spending far less search compute, since the oracle requires training dozens of drafters at 70B.
Why not just…?
Why not just search wirings directly on the target model? At 8B this is a real threat. Training about 30 drafters is affordable, and if direct search is cheap the proxy adds nothing there. Our claim lives at 70B and above, and under repeated domain shift, where re-searching per model and per domain multiplies cost. The oracle experiment settles this. We report the gap between the profile-chosen wiring and the oracle, together with the compute each required. If the gap is large at 70B, the proposal fails on its own terms.
Why not just keep Gumiho's 2+5 split everywhere? It may be that one fixed front-loaded split is near-optimal for every domain and scale, in which case wiring is a solved hyperparameter. AngelSpec's need for separate chat and code/math drafters is evidence against one-size-fits-all, but it is not direct evidence about the split itself. Gumiho is in our candidate population, so if it sits at the top for every domain and scale we will report that wiring choice does not matter beyond front-loading.
Why not just read dependencies from the target's attention, as DAPD does? It requires no proxy model and no Monte Carlo sampling. DAPD itself notes that attention reflects how information is routed through the network rather than probabilistic dependence. Our attention-derived ablation compares the two signals head to head. It measures which one better predicts per-slot acceptance gains, and which one picks better wirings.
Why not just filter draft tokens by confidence? Confidence filtering can be stacked on top of any drafter, so it is complementary rather than an alternative to choosing a wiring. It also cannot catch the "San York" case, where both slots are individually confident. We will apply the same confidence-based pruning to every drafter so it does not confound the comparison.
Proposed Experiments
Datasets. Evaluation uses GSM8K, MATH500, HumanEval, MBPP, and MT-Bench for chat. These are the benchmarks Gumiho and AngelSpec report on.
Models. Targets are LLaMA-3 8B and 70B, with 1B and 3B proxies from the same tokenizer family.
Metrics. - Mean accepted length. - Wall-clock speedup over plain decoding. - Tokens per second at batch size 1 and at moderate batch sizes. - Rank correlation between the proxy-predicted ordering of wirings and their measured ordering on the target.
Decisive check 1: do profiles transfer across scale? Kendall rank correlation of the weighted profile between proxy and target must be high on each domain. If profiles shift with scale, the transferability premise is false and the rest is moot.
Decisive check 2: does the profile predict which wiring wins? Across the 30 trained wirings, the proxy-derived score must rank-correlate with measured accepted length on the 70B target clearly better than the attention-derived and random scores do.
Decisive check 3: does the profile track domain shift? We swap domain, from chat to code. The re-profiled wiring should beat both the old domain's wiring and Gumiho on the new domain. If wirings chosen for different domains perform the same, we will conclude that dependency structure matters less than AngelSpec implies.
Sanity check: is the estimator accurate? Because conditional mutual information is estimated by Monte Carlo, we first validate the estimator on the synthetic Reed-Solomon benchmark from Adaptation to Intrinsic Dependence in Diffusion Language Models. There the dependence between tokens is known in closed form, so recovered per-slot terms can be checked against ground truth.
References
- Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding
- AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding
- On the Learning of Non-Autoregressive Transformers
- PLoP: Precise LoRA Placement for Efficient Finetuning of Large Models
- On Network Design Spaces for Visual Recognition
- Decoding Speculative Decoding
- Fast and Expressive Multi-Byte Prediction with Probabilistic Circuits
- DAPD: Dependency-Aware Parallel Decoding via Attention for Diffusion LLMs
- Adaptation to Intrinsic Dependence in Diffusion Language Models