Hyperbolic Token Depth as a Safety Signal for Loose Speculative Decoding

Speculative decoding speeds up large language models by letting a small "draft" model guess several tokens ahead. The large "target" model then checks all of them in one pass. Standard checking accepts a guess only if it matches what the target would have produced, so a harmless synonym ("huge" for "large") throws away the rest of the draft. Recent "loose" verifiers accept some mismatches. Some use the target's uncertainty, some use embedding similarity, and some use small trained classifiers. They share a blind spot. The tokens whose swaps are most dangerous, such as the digit "7" for "8", one variable name for another, or one city for another, are often the tokens that look most alike to a similarity measure. They belong to the same fine-grained category, and that is exactly why exchanging them changes the answer.

We propose treating the vocabulary as a hierarchy, embedded in a small hyperbolic space. Hyperbolic space is a curved geometry in which available room grows exponentially away from a center, so trees fit into it with little distortion. In such an embedding, generic tokens sit near the center and specific tokens sit near the edge. The distance between two siblings also grows with their depth. Two interchangeable function words end up close together, while two digits or two proper names end up far apart even though they are "siblings." That is the behavior a safe acceptance rule needs. We want to test whether this one geometric quantity, computed cheaply for the few candidate tokens at each step, separates harmless from harmful mismatches better than flat similarity measures. A second question is whether it can act as a training-free acceptance signal on top of existing loose verifiers.

Core Claim

The claim is that the danger of a token substitution is mostly about specificity. A flat Euclidean or cosine similarity cannot express specificity. A hyperbolic embedding expresses it as radius, and distance grows with radius. Concretely, we fit a low-dimensional (16–32 dimensional) Poincaré-ball embedding of the target model's vocabulary. The Poincaré ball is a standard model of hyperbolic space. The embedding is fit to preserve how often two tokens appear together in the target's top-$k$ predictions on unlabeled text. This co-occurrence is an unsupervised proxy for interchangeability.

Tokens that are plausible alternatives in many contexts should land near the origin. Tokens that are interchangeable only with a few near-duplicates should land near the boundary. For a draft token $y$ that mismatches the target's top token $x$, we accept $y$ when three conditions hold. First, $y$ is in the target's top-$k$. Second, the target's entropy exceeds a floor. Third,

$$d_{\mathbb{B}}(x,y) = \operatorname{arcosh}\!\left(1 + \frac{2\|x-y\|^2}{(1-\|x\|^2)(1-\|y\|^2)}\right) \le \tau ,$$

where $x, y$ here denote the embedded points, $\|\cdot\|$ is the Euclidean norm inside the unit ball, and $\tau$ is a single threshold. The denominator makes the same Euclidean gap count for far more near the boundary. This is the mechanism that penalizes swaps among specific tokens without any hand-built list of "dangerous" token types.

A secondary use of the same geometry addresses an open problem noted across the relaxed-verification literature: the lack of any sequence-level limit on accumulated drift. Each accepted mismatch spends $d_{\mathbb{B}}(x,y)$ from a per-response budget $B$. Once the budget runs out, verification reverts to exact matching.

Relation to Prior Work

Three papers supply the evidence for the gap. ARC-Decode reports that over 70% of rejected draft tokens produce continuations consistent with the target's trajectory, so relaxing verification is worthwhile. Its Local Tolerance Score mixes Euclidean embedding similarity with logit gaps under a Jensen–Shannon bound. That score is the closest existing design to ours, and it is the component we replace. Nothing in ARC-Decode's score encodes specificity, so digit and entity swaps receive high similarity.

FLy avoids trained verifiers by combining an entropy gate with a "wait-and-see" window that watches whether the target self-corrects. Its entropy signal is a property of the context, not of which two tokens are being swapped. Our signal is complementary and can sit inside FLy's gate.

AutoJudge and SelfJudge train linear classifiers on target hidden states from mined labels. They are contextual and strong, but they need task-specific label mining and can degrade out of distribution. We use their labeling procedures only to evaluate our score, never to train it.

On the geometry side, the method comes from Hyperbolic Neural Networks, which defines Poincaré-ball operations and shows hyperbolic classifiers win most in low dimensions. Hyperbolic Large Language Models motivates the premise: it reports that pretrained language models show "intrinsic hyperbolicity," with frequent generic tokens near the origin and rare specialized tokens toward the boundary. That survey never touches decoding, and we turn its observation into an acceptance rule.

One claim from the same literature constrains the design. Hyperbolic advantages shrink at high dimension ($D \ge 256$), because Euclidean space then has enough spare room. For that reason we do not propose replacing the 4096-dimensional hidden-state judge with a hyperbolic classifier; that idea is the obvious one at this intersection and would probably lose to a Euclidean multilayer perceptron. We deliberately work in a tiny embedding, where the geometry matters, and we include a high-dimensional Euclidean control.

The embedding is also unrelated to the tree-shaped draft methods of Medusa and tree attention (Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding). Those methods organize candidate sequences, not the vocabulary.

Research Program

The first stage builds the embedding. We run the target model (Llama-3.1-8B, then Llama-3.1-70B) over a generic unlabeled corpus disjoint from all evaluation sets. At each position we record which tokens appear jointly in the top-$k$. We then fit Poincaré embeddings with a standard distance-ranking loss, so that frequently co-occurring pairs are close. This is done once per tokenizer and model. At inference the cost is one 16-dimensional distance per candidate pair, which is negligible next to a target forward pass. It also sidesteps the concern about missing hardware kernels for hyperbolic operations, because nothing hyperbolic runs inside the network.

The second stage is a diagnostic study of mismatches. We use AutoJudge's substitution-and-check procedure on GSM8K and HumanEval-style tasks, where correctness is verifiable. We use SelfJudge's Bayes-factor labels on MT-Bench-style prompts. Together these label a pool of draft/target mismatches as harmful or harmless. We then measure how well each scalar score ranks harmful mismatches above harmless ones, reporting the area under the ROC curve (AUROC). The comparison is broken down by token type: numerals, identifiers, named entities, and function words.

The third stage integrates the score into decoding. We test it on its own with top-$k$ and entropy gating, inside FLy's deferred window, and with the sequence budget $B$. The same threshold and budget are used across all tasks, so we can judge how much calibration the method needs.

Baselines

The naive reference is standard lossless speculative decoding with the same draft/target pair and exact matching, run in vLLM on the same GPUs. Every speedup is reported relative to it. Standard decoding without speculation gives the accuracy ceiling.

The strongest training-free competitor is FLy. It reports a 2.81× average speedup on Llama-3.1-70B while recovering at least 99% of base accuracy. The research file gives no isolated GSM8K or HumanEval number for it, so we rerun it ourselves. The fairness rules are the same draft source, tree size, entropy thresholds tuned on the same held-out prompt set, and the same tuning budget as ours. Beating FLy means one of two outcomes. Either we match its accuracy recovery at a higher acceptance length, or we reach its speedup with fewer accuracy failures on numeral- and entity-heavy items. FLy with our distance added to its gate is also reported, since the two signals may simply combine.

ARC-Decode is the most direct competitor. It reports up to 1.6× additional speedup over EAGLE-3 and an MT-Bench score of 7.02 vs. 6.77 on Llama-8B. We run its Local Tolerance Score with its published calibration (200 OASST1 prompts) and replace only the embedding-similarity term with our hyperbolic distance. This isolates the geometric ingredient. Beating it means a higher acceptance rate at equal or better accuracy on the diagnostic subsets.

The trained judges set an upper reference, not a target we expect to beat in-domain. AutoJudge reports about 2× speedup over standard speculative decoding on GSM8K with Llama-3.1-70B, with at most 1% math-accuracy drop and 107.4 tokens/second for the 8B/70B pair on A100s. SelfJudge reports a 23.5% throughput increase on GSM8K for Llama-3.1-8B, within 1% accuracy. We retrain both on their original task data and then evaluate them out of distribution on MGSM and MBPP. There our training-free score should hold up where they degrade.

Three ablations isolate the key ingredient. The first fits a Euclidean embedding to the same co-occurrence data at matched dimension, 16. The second fits the same Euclidean embedding at 256 dimensions, which tests whether any win is really about curvature. The third uses raw cosine similarity of the target's output-embedding rows, which is ARC-Decode's style of signal.

Why not just…?

Why not just weight cosine similarity by inverse token frequency? This is the most serious threat. The radius of our embedding may end up tracking frequency, and a cosine-times-IDF score would then capture most of the benefit with no geometry at all. The diagnostic study includes this score as a first-class comparison, with its weight tuned on the same held-out set. If it matches hyperbolic distance on AUROC for the numeral and entity subsets, we will report that the useful content is specificity, not curvature, and ship the simpler rule. Our reason to expect a gap is structural. Frequency is a per-token property. The hyperbolic metric combines depth with the pair's distance, so two moderately frequent but interchangeable tokens can stay close while two rare synonyms still pass.

Why not just train a better judge on hidden states, for example a small multilayer perceptron? In-domain, a contextual judge probably wins, and we say so. The costs are the ones the file documents. Label mining took 82 hours for 14k labels in AutoJudge. It needs verifiable tasks. Trained judges also degrade out of distribution. The comparison that matters is out of distribution (MGSM, MBPP, MT-Bench after training on GSM8K). We also test the pairing, feeding our distance as an extra feature to SelfJudge's classifier. If the feature adds nothing, that shows the hidden states already encode specificity.

Why not just rely on entropy gating and self-correction, as FLy does? Entropy says when the target is unsure. It does not say which of its candidates are safe to exchange. In a high-entropy step over digits, entropy gating defers exactly the swaps that corrupt math answers. The per-type breakdown shows whether FLy's failures concentrate on specific-token swaps. If they do not, the added value of our signal is small, and the combined FLy-plus-distance run will show it.

Proposed Experiments

The decisive check comes first and costs little. On the labeled mismatch pools, hyperbolic distance at 16 dimensions must exceed all three ablations and cosine-times-IDF in AUROC, specifically on numeral, identifier, and entity mismatches. If it does not, the decoding experiments are not worth running.

If it passes, we run end-to-end decoding on GSM8K, MATH-500, HumanEval, MBPP, LiveCodeBench, and MT-Bench, with MGSM as an out-of-distribution test. We report mean accepted length, wall-clock speedup against exact speculative decoding, and accuracy relative to the target alone. The main pairs are Llama-3.1-8B/70B and a smaller draft/Llama-3.1-8B pair.

A second check targets drift. We compare accuracy against response length with and without the sequence budget $B$ on long MATH-500 derivations. The budget should flatten the accuracy decline on long outputs without giving back most of the speedup.

A third check measures sensitivity to calibration. We fix $\tau$ and $B$ from one domain and apply them unchanged to all others, then report how much speedup and accuracy move. This directly tests whether the context-free token geometry reduces the per-distribution calibration that ARC-Decode and the trained judges require.