Speculative decoding speeds up a large language model by letting a small "draft" model guess several tokens ahead while the large "target" model checks them all in one parallel pass. Standard checking accepts a guess only if it matches what the target would have produced. Recent methods relax this rule and accept guesses that differ in wording but not in meaning, which roughly doubles the speedup. The open problem is that each relaxed acceptance is judged only locally. Nobody measures whether one harmless-looking substitution, say "large" in place of "huge", or a different but equivalent algebra step, causes the target to disagree again a few tokens later, and whether those new disagreements then cause more. We propose to measure this propagation directly as a branching ratio: the expected number of new disagreements that one accepted substitution induces downstream. We then build a verifier that keeps this ratio below one. It combines two things. The first is exact checking restricted to a small set of protected quantities in the trace (numbers, equations, identifiers), which we call the ledger. The second is an online controller that tightens relaxation for everything else whenever the estimated ratio approaches one.
The design comes from two results in mathematical physics, used as a lens rather than as imported theorems. Natural Boundaries for Scattering Amplitudes shows how a recursion (elastic unitarity) turns one singularity into infinitely many. Once nothing isolates the elastic region, the singularities pile up into a wall past which the scattering amplitude cannot be continued. In gapped theories, the mass gap is the buffer that keeps each singularity isolated. Seiberg duality in matrix models II shows that when masses go to zero, matrix integrals diverge along unconstrained "flat directions." The correspondence with gauge theory survives only if explicit delta-function constraints restrict integration to the space of physically distinct configurations. The translation we test has three parts. Relaxed acceptance is a recursion. The branching ratio plays the role of the gap. The ledger plays the role of the delta-function constraint that pins down the few quantities the final answer actually depends on.
Core Claim
Treat relaxed acceptances as the "immigrants" of a branching process: each one independently spawns a random number of later disagreements with mean $b$, and each of those spawns more in turn. Let $\varepsilon$ be the per-token rate at which relaxed substitutions are accepted and $n$ the trace length. Standard branching-process algebra, which sums the geometric series of descendants, gives the expected total number of positions where the relaxed trace departs from what the target alone would have written:
$$\mathbb{E}[D_n] \approx \frac{n\,\varepsilon}{1-b} \quad (b<1),$$
with growth faster than linear once $b \ge 1$.
Existing per-token bounds effectively assume $b=0$. Examples are the Jensen–Shannon bound in ARC-Decode and the entropy gate in FLy. Both of those papers concede that global, sequence-level drift is not controlled.
Our claim has three parts, each testable:
- $b$ is measurable at low cost.
- $b$ varies sharply across contexts. It is small in free-form prose and approaches or exceeds one when a substitution touches a quantity that later steps reuse.
- Pinning ledger tokens exactly and adapting tolerance elsewhere keeps $b$ bounded away from one. This should retain most of FLy's speedup while removing the "insidious" failures: cases where a relaxed substitution never causes visible divergence but still changes the final answer.
Research Program
Measuring the branching ratio. On a calibration set, run relaxed decoding and record every accepted substitution. For each one, fork the trace at that position. In one branch, continue from the target's own token under strict speculative decoding. In the other, continue from the substituted token. Count the new target-draft disagreements inside a window of $w$ tokens. Averaging these counts gives $\hat b$ per context type.
We will also fit a lightweight predictor of $\hat b$ from quantities already available at verification time. These are the target's entropy, the substitution's embedding distance (reusing ARC-Decode's Local Tolerance Score), and whether the substituted span touches a ledger quantity. The deployed controller uses the predictor, not the expensive forks.
Building the ledger. For math, a regex-plus-SymPy parser extracts numerals, variable bindings and equations from each step and puts them in canonical form. For code, an incremental tokenizer marks identifiers, literals and operators. Ledger tokens must match the target exactly, or pass the standard probability-ratio test under sampling. A cheap consistency pass also re-evaluates newly asserted equalities against earlier ledger entries.
This is the analogue of the chiral-ring observation in Chiral Rings and Anomalies in Supersymmetric Gauge Theory: there, the low-energy answer depends only on a small protected set of operators, and every other variation is irrelevant. We do not claim an algebraic theorem for language. We claim an empirical decomposition: final-answer changes should be explained almost entirely by ledger perturbations.
Controlling relaxation. For non-ledger tokens, the acceptance threshold $\tau_t$ is updated online so that the running predicted $\hat b$ stays below $1-\delta$. We also track a global budget $n\varepsilon/(1-\hat b)$ against a user-set drift tolerance. The same analogy suggests a sharp, non-smooth change in behaviour at $b=1$, much as The strong couplings of massive Yang-Mills theory finds a non-smooth massless limit. We will test this rather than assume it, by sweeping a fixed relaxation threshold and looking for a knee in final-answer agreement as $\hat b$ crosses one.
Baselines
The naive reference point is strict speculative decoding with an EAGLE-3 style draft. It is lossless by construction, and Scaling Speculative Decoding with Lookahead Reasoning reports it capped at 1.4x on GSM8K. Target-alone greedy decoding supplies reference answers and traces for every agreement metric.
The strongest existing competitor is FLy. It is training-free, gates relaxation by target entropy, and uses a deferred window to catch divergence. It reports an average 2.81x speedup on Llama-3.1-70B with at least 99% of target accuracy retained. We will run FLy with its released thresholds on the identical Llama-3.1-70B target and 8B draft, the same hardware and the same prompts. We will also give it a tuning budget equal to ours: the same number of calibration prompts, used to sweep entropy threshold and window length.
ARC-Decode is the strongest method with a stated risk bound. It reports up to 1.6x over EAGLE-3 and a rise in accepted tokens per cycle from 4.01 to 5.28 on Vicuna-13B. We will calibrate it on its 200 OASST1 prompts plus our calibration split.
SelfJudge is the strongest trained verifier. It reports 23.5% higher throughput on GSM8K and accuracy within 1%. We will retrain its logistic judge on our target's hidden states with its self-supervised labels.
SpecReason is the step-level alternative. It reports 1.4x–3.0x speedup, and its utility threshold will be tuned on the same calibration split.
Two ablations isolate our key ingredients: the ledger with a fixed relaxation threshold (no controller), and the controller without the ledger.
Beating FLy means one of two outcomes. Either we match its speedup within 10% while cutting the rate of final answers that differ from target-alone decoding by at least half on AIME '24 and MATH-500. Or, at equal answer-change rate, we deliver higher speedup. For the insidious-error rate itself, no reported result found on Lacuna for any baseline.
Why not just…?
Why not just sum the per-token Jensen–Shannon bounds into a global budget? Summing local bounds is exactly the $b=0$ special case of our equation. It is accurate only if a substitution never changes what the target does later. The forking measurement tests this directly. If measured $\hat b$ stays near zero across contexts, the cumulative-JS budget suffices and our controller adds nothing. We will report that outcome as a negative result.
Why not just lengthen FLy's deferred window? This is the most serious threat. A longer window catches more propagated disagreements, and FLy already reaches at least 99% accuracy. Its weakness, acknowledged in the file's own analysis, is that it detects only visible divergence. A substituted coefficient that the target then consistently builds on produces no divergence at all, only a wrong answer. We will sweep FLy's window up to our fork length $w$ and measure the answer-change rate and speedup at each setting. If a long-window FLy matches our answer-change rate at similar speed, the ledger is unnecessary.
Why not just use a process reward model or a trained judge? SpecReason's step scores and SelfJudge's classifier judge plausibility. They do not track whether earlier quantities are reused consistently. They also add verifier compute and, for SpecReason, a second KV cache. We compare at matched wall-clock time. If SelfJudge catches ledger errors as well as exact ledger matching does, the ledger reduces to a cheaper but equivalent heuristic, which we would report as such.
Proposed Experiments
The main testbed is Llama-3.1-70B as target with an 8B draft. It runs on GSM8K, MATH-500, AIME '24, HumanEval, LiveCodeBench and, as an out-of-distribution check, MGSM. A second pair uses a 1.5B draft for a 32B reasoning target, following Lookahead Reasoning's setting, to stress traces thousands of tokens long.
Metrics are:
- wall-clock speedup;
- accepted tokens per cycle;
- task accuracy;
- final-answer agreement with target-alone decoding;
- an insidious-error rate, defined as answer changes with no disagreement flagged inside the deferred window.
Three checks are decisive:
- Fork-measured $\hat b$ must vary across context types by at least a factor of three. Otherwise the core premise fails.
- Sweeping a fixed relaxation threshold should show answer-change rate rising sharply near $\hat b \approx 1$, not linearly in $\varepsilon$.
- Among all answer changes under ledger-free relaxation, the fraction traceable to a ledger-token substitution should exceed 80%. This is what justifies restricting exact checks to the ledger.
If all three hold, and the full system meets the bar against FLy set out in the Baselines section, the branching-ratio view is a useful account of global drift in relaxed verification.
References
- Natural boundaries for scattering amplitudes
- Seiberg duality in matrix models II
- ARC-Decode: Accelerated Decoding with Risk-Bounded Acceptance
- Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Match
- Chiral Rings and Anomalies in Supersymmetric Gauge Theory
- The strong couplings of massive Yang-Mills theory
- Scaling Speculative Decoding with Lookahead Reasoning