Reading the Draft Tree's Shape from a Single Head Output

Speculative decoding speeds up a large language model by guessing several future tokens cheaply and then checking all the guesses in one pass of the full model. The best current systems organize the guesses as a tree of alternative continuations. They decide how wide and deep that tree should be by running a small search at every step: they multiply draft probabilities along paths, expand the most promising branches, and prune the rest. That search costs time. It also tends to build large trees on steps where most branches will be thrown away, and those wasted branches become expensive once the server is batching many requests. We propose to remove the search entirely for drafters that live inside the target model. A lookahead head would emit, for each future position, a point in hyperbolic space. Hyperbolic space is a curved geometry where the room available grows exponentially with distance from the center, so it fits trees naturally. The point's direction picks the candidate tokens. Its distance from the center says how committed the head is, and that distance sets how many branches the tree gets at that depth. The whole tree shape is then a lookup, fixed before any per-path scoring is done.

The appeal is practical. Self-drafting heads such as Medusa avoid hosting a second model, but they verify a fixed tree. Dynamic-tree methods such as EAGLE-2 and OPT-Tree adapt the tree, but they pay for a separate drafter and a search at every step. A head whose own output geometry encodes "how branchy is the future here" would give adaptive trees at the cost and simplicity of Medusa. The idea is also testable in an unusually clean way. If an ordinary Euclidean head with a learned scalar does just as well, the geometry is unnecessary, and the experiments are designed to say so.

Core Claim

The claim has two parts, and they can fail independently. The first part is architecture-agnostic. A self-drafting head can predict the per-depth branching budget of the speculation tree in one shot from the backbone's hidden state. That one-shot budget can match the acceptance length of step-wise greedy tree expansion while cutting drafting overhead and the number of tokens sent to verification. The saving in verified tokens matters most at larger batch sizes, where Hydra reports that tree speculation gains decay as verification becomes compute-bound.

The second part is geometric. The best signal for that budget is the radius of a hyperbolic head output rather than a separately trained Euclidean scalar. In a hyperbolic classifier, a point near the center is nearly equidistant from many class boundaries, and its token distribution is spread out. As the point moves outward, distances to sibling classes separate exponentially, and the distribution concentrates. The same quantity therefore sets both "which tokens" and "how many to keep," so the two cannot drift out of agreement.

Learning the Predictability of the Future is the direct motivation for the geometric part. On video action forecasting it showed that a hyperbolic predictor hedges near the center when the future is ambiguous and moves outward as the future resolves. In that setting a 64-dimensional hyperbolic embedding beat a 256-dimensional Euclidean one. Nobody has asked whether the same hedging behavior, applied to next-token lookahead, is a usable control signal for speculation.

Research Program

We will build hyperbolic lookahead heads on a frozen backbone, in the Medusa-1 setting. Head $k$ maps the final hidden state $h_t$ to a point $z_k$ in the Lorentz model of hyperbolic space. Its vocabulary logits come from signed distances to per-token hyperplanes. For the linear layers we will use the distance-to-hyperplane construction from Fast and Geometrically Grounded Lorentz Neural Networks, for two reasons. That paper shows earlier Lorentz layers grow output norms only logarithmically with training steps, which would cripple a radius-based signal. It also shows that caching weights after training brings hyperbolic inference latency close to an ordinary matrix multiply. The older Möbius operations of Hyperbolic Neural Networks serve only as a stability fallback; their numerical difficulties are what constrain us away from the Poincaré ball.

Training uses ShareGPT (dataset) with the usual cross-entropy on target-model tokens at offsets $1$ to $K$. We add one auxiliary term that ties the head's radius $r_k = d(o, z_k)$, its geodesic distance from the origin, to the negative entropy of the target model's own next-token distribution at that offset. This term also gives the second part of the claim a clean comparison: every alternative signal gets the same supervision.

At inference the tree is built by a fixed rule. For each depth $k$ we compute a width $w_k$ from the radius:

$$w_k = \mathrm{clip}\big(\lfloor W_{\max}\, e^{-\alpha r_k} \rfloor,\ 1,\ W_{\max}\big), \qquad \text{subject to } \sum_{k=1}^{K} \prod_{j\le k} w_j \le N.$$

Here $W_{\max}$ is the maximum branching factor, $\alpha$ is a single scalar calibrated offline on held-out data, and $N$ is the node budget. The exponential form follows the exponential separation of sibling classes with radius. If the budget is exceeded, depths are truncated from the deepest end. This follows the Monotonic Rule of Multi-Candidate Speculative Decoding, which found that spending candidate branches on early positions maximizes expected acceptance. The top-$w_k$ tokens at each depth then form a Cartesian tree, which is verified with Medusa's tree-attention mask. The only work done at each step is $K$ head evaluations and one sort per depth. There are no path-probability products, no priority queue, and no reranking pass.

Verification is unchanged, so greedy decoding stays lossless. For sampling we use standard speculative rejection sampling. Following the frontier analysis in A Theoretical Perspective for Speculative Decoding Algorithm, our method changes only where the method sits between drafting cost and acceptance. It does not change the output distribution.

The analysis has three layers. The first is a calibration study: how well does each candidate signal predict the realized accepted depth at each step? The second is end-to-end speed. The third is a regime study on the tasks where speculation is known to struggle: summarization and retrieval-augmented generation in SpecBench, where Hydra saw smaller gains, and the distribution-shifted translation setting where Multi-Candidate Speculative Decoding found linear drafts fragile. These are the steps where shrinking the tree should save the most verification compute.

Baselines

The naive reference point is vanilla autoregressive decoding on the same backbone and hardware. Every speedup is reported against it.

The first speculative baseline is Medusa with its static tree (Medusa). It is the direct parent of our design: the same frozen backbone, the same training data, and the same head count, differing only in having a fixed tree. Medusa reports over 2.2x speedup for MEDUSA-1 and 2.3x–2.8x for MEDUSA-2 on MT-Bench. Medusa heads for a Vicuna-7B model train in about 5 hours on one A100, and we will give our heads the same budget. Beating Medusa means higher wall-clock speedup at an equal or smaller average verified-node count.

The second baseline is Hydra++, run from the released code (zankner/Hydra) (Hydra). It is the strongest self-drafting competitor. It reports up to 2.70x throughput over autoregressive decoding, up to 1.31x over Medusa, and an average 1.24x over Medusa on SpecBench. It buys those gains with sequentially dependent heads and a prefix-attention layer. Our method keeps independent heads and adapts the tree instead. Hydra++ is therefore the test of whether tree shape or head dependence is the better use of a fixed latency budget. We will also run a combination, Hydra heads with our radius-shaped trees, because the two ideas need not compete.

The strongest existing method overall is EAGLE-2. It reports 3.05x–4.26x speedup over vanilla decoding, about 2x over Medusa, and acceptance lengths often reaching 4–5.5 tokens per pass. It uses a trained external draft model and an expansion-plus-reranking search at every step. We do not expect to beat it on raw speedup at batch size one with a frozen backbone, and we will not claim to. Instead we will run it on the same hardware at batch sizes from 1 upward, matching node budgets. Success against EAGLE-2 means recovering most of its batch-size-one speedup with no auxiliary model. It also means overtaking it as batch size grows, because we verify fewer nodes when the future is uncertain.

OPT-Tree is a secondary point. It reports a mean acceptance length of 7.74 versus 4.97 for EAGLE on LLaMA-2-70B, and over 10 tokens per step at 500 nodes. It shows what large adaptive trees can reach when verification is cheap. We will include it in the batch-size sweep to expose the node-budget tradeoff.

The ablations that isolate the key ingredient all share the same heads, training data, node budget, and tuning budget for $\alpha$ or its equivalent. The first is a Euclidean head whose hidden-state norm drives the same width rule. The second is a Euclidean head with a small scalar output trained on the same entropy target. The third is a Euclidean head whose widths come from an EAGLE-2-style greedy expansion over its own path probabilities. The hyperbolic part of the claim holds only if the hyperbolic radius beats the first two on calibration and on speedup. The one-shot part holds only if any one-shot variant matches the third on acceptance length while drafting faster.

Why not just…?

Why not just make the Euclidean heads bigger, as Hydra++ does? Hydra's own results show that deeper heads and a prefix-attention layer raise throughput. They also add latency to every draft step, and they leave the tree static, so wasted branches on uncertain steps remain. The dimension argument from Hyperbolic Neural Networks and the 64-versus-256 result from Learning the Predictability of the Future suggest hyperbolic heads could stay small. That evidence comes from taxonomies and video, though, not from token prediction. We treat it as a hypothesis. The Hydra++ comparison and the combined variant settle whether head capacity or tree shape is the bottleneck.

Why not just run EAGLE-2's dynamic expansion on top of Medusa heads? This is the most serious threat. The heads already produce probabilities, and the greedy expansion could plausibly recover most of the adaptive benefit. The question is whether the per-step search costs enough to matter. OPT-Tree needs an efficiency threshold precisely because expansion overhead can cancel acceptance gains. Our third ablation implements exactly this alternative and reports drafting milliseconds per step next to acceptance length. If greedy expansion on Medusa heads matches our acceptance length at negligible overhead, the one-shot claim fails and we will report that.

Why not just train a Euclidean scalar that predicts the tree width? A learned scalar on the same entropy target could do everything the radius does, and the Euclidean hidden-state norm already scales logit sharpness. This is why both appear as ablations with identical supervision. The geometric part of the claim survives only if the hyperbolic radius is better calibrated to realized acceptance depth, especially at deeper offsets where hierarchy should matter. If it is not, the contribution reduces to one-shot shaping with a cheap Euclidean predictor. That is still a useful result, but it would mean the hyperbolic direction contributes nothing here.

Why not just use LayerSkip and avoid heads entirely? LayerSkip shares weights and the KV cache and reports 2.16x on CNN/DM, 1.82x on HumanEval, and 2.0x on TOPv2. However, it drafts a single linear sequence, so one early rejection discards the rest of the draft. It also requires retraining the backbone with layer dropout, whereas our heads leave the backbone frozen. We will report LayerSkip's published numbers alongside ours on HumanEval and CNN/DM as context rather than as a matched baseline, because the training regimes differ.

Proposed Experiments

The backbone is Vicuna-7B, matching the Medusa and Hydra setups, with a 13B check if results hold. Evaluation covers MT-Bench (Judging LLM-as-a-judge with MT-Bench and Chatbot Arena), SpecBench, GSM8K, and HumanEval. For GSM8K and HumanEval we found no per-task speedup reported on Lacuna for EAGLE-2 or OPT-Tree; both report only aggregates. We will therefore run all baselines ourselves on those tasks. Metrics are wall-clock speedup over vanilla decoding, mean acceptance length, verified nodes per accepted token, drafting time per step, and throughput at batch sizes 1, 4, 16, and 32. Output quality is checked by exact match under greedy decoding and by MT-Bench score under sampling.

The first decisive check is calibration, measured before any speed experiment. For each step we record the realized accepted depth. We then compute rank correlations between that depth and each of the following: the hyperbolic radius, the Euclidean norm, the learned Euclidean scalar, and EAGLE-2-style path-probability products. If the radius is not at least as predictive as the path products, the one-shot design cannot work, and we stop there.

The second check is the matched-budget speed comparison. Given equal node budgets, the claim succeeds if radius-shaped trees beat static Medusa trees and match greedy-expanded Medusa trees in acceptance length, with lower drafting time.

The third check is the batch-size and high-entropy regime: SpecBench summarization and retrieval-augmented generation, and WMT English–German. Here the prediction is specific. Radius-shaped trees should shrink on high-entropy steps, so verified nodes per accepted token should fall relative to EAGLE-2 and OPT-Tree. That should translate into a throughput lead that widens with batch size. If none of these three effects appears, the hyperbolic hedging account does not carry over from video forecasting to token speculation, and that negative result would itself be worth stating plainly.