Reasoning language models get better answers by writing longer chains of thought. Every token they write, though, adds an entry to the key-value (KV) cache, the per-token memory of attention keys and values that the model re-reads at every step. On fixed hardware, the usual way to fit longer traces is to store that cache at 4 or 8 bits instead of 16. Deployments currently set both choices uniformly: one bit width for every attention head, and one token budget for every query. This note proposes a label-free, analytical way to set both. We borrow the Bussgang decomposition from low-resolution wireless receivers. It rewrites any quantizer's output as a scaled copy of its input plus a noise term that is uncorrelated with that input. Applied to quantized keys, it predicts the length at which quantization noise begins to scramble attention. Past that length, extra reasoning tokens stop helping.
The central observation is simple. Quantizing a key does two different things to an attention score. It shrinks the score by a gain factor, which acts like a temperature change and can be undone with one scalar per head. It also adds noise that behaves like random jitter on every score. The damage from the jitter is not fixed; it grows with the number of cached tokens competing for attention. Uniform bit allocation ignores the first effect. It also ignores the fact that the second effect interacts with the token budget. Mean-squared quantization error, the usual sensitivity measure, mixes the two together.
Core Claim
Consider a key $k$ stored with quantizer $Q$. Write $Q(k) = \alpha k + e$, where $\alpha = \mathbb{E}[k^\top Q(k)]/\mathbb{E}[\|k\|^2]$ and $e$ is the remainder. This choice of $\alpha$ makes $e$ uncorrelated with $k$. The decomposition exists for any input distribution, including the heavy-tailed, outlier-rich activations of real transformers. Only the closed-form expression for $\alpha$ needs Gaussian inputs, so we estimate $\alpha$ and the distortion variance $\sigma_e^2$ per head from a few hundred unlabeled calibration traces.
The attention score of a query $q$ on cached token $j$ then becomes $\alpha s_j + \eta_j$. Here $s_j$ is the full-precision score, and $\eta_j = q^\top e_j/\sqrt{d}$ sums $d$ roughly independent products. By the same averaging argument that makes aggregate array distortion approximately Gaussian in massive MIMO (Wu et al.), $\eta_j$ is approximately Gaussian with variance $\sigma_\eta^2 = \|q\|^2\sigma_e^2/d$.
Suppose a head must retrieve one relevant token among $n$ cached tokens, and the full-precision score of the relevant token exceeds the largest competitor by a margin $m$. Retrieval then survives quantization roughly when
$$\alpha\, m \gtrsim \sigma_\eta \sqrt{2\ln n},$$
where the $\sqrt{2\ln n}$ factor is the typical maximum of $n$ independent Gaussian noise draws. Dividing out $\alpha$ leaves a per-head ratio $\rho = \alpha m/\sigma_\eta$. From that ratio we obtain a critical length $n^\ast = \exp(\rho^2/2)$, beyond which noise-induced retrieval errors dominate.
The testable claim has three parts. First, $n^\ast$, aggregated over the heads that do long-range retrieval, predicts where a quantized reasoning model's accuracy-versus-token-budget curve flattens. Second, correcting the gain $\alpha$ recovers part of the gap between 4-bit and 16-bit caches at no memory cost. Third, a memory allocation that assigns bits to heads by $\rho$ and caps each query's token budget by its running $n^\ast$ beats uniform bits with a uniform budget at equal memory.
Relation to Prior Work
The motivating evidence comes from Not All Bits Are Equal. That paper swept about 1,700 configurations of weight and KV precision against token budgets and parallel samples. It found a scale-dependent threshold: small models should spend memory on precision, and larger ones on longer traces. It also found orderings such as an "8B model at 8-bit consistently outperforms a 14B model at 4-bit." That study is empirical and model-level, and gives no formula for where the crossover sits. We aim to predict its crossovers from activation statistics and to refine its uniform-per-model precision to per-head precision.
The method comes from the Bussgang literature. Linearizing a hard quantizer into gain plus uncorrelated distortion is standard for 1-bit and low-resolution converters, both in channel estimation (Jacobsson et al.) and in precoding (Jacobsson et al.). There it yields signal-to-distortion ratios that guide system design. Thrampoulidis et al. showed that high-dimensional estimation from non-linear measurements behaves exactly like estimation from linear measurements with gain $\mu$ and added Gaussian noise $\sigma$. They also showed that the Lloyd-Max quantizer minimizes downstream error. We use the same gain-plus-noise reduction, but the downstream task here is softmax retrieval over a growing cache rather than LASSO recovery.
Two results constrain the design. Fesl et al. show that linear Bussgang estimators fall short of optimal ones under non-Gaussian priors. For that reason we use the decomposition only to predict noise levels, not to build an estimator, and we check the Gaussian-noise assumption directly. Guichemerre and Studer show that real converter non-idealities call for affine rather than purely scalar corrections. Accordingly, we test an offset term alongside $\alpha$, since integer KV quantizers with zero points may need one.
On the allocation side, Damani et al. learn per-query marginal-reward predictors and halve samples relative to uniform best-of-$k$. Thought calibration (Wu et al.) trains probes and uses conformal risk control to stop reasoning early. Both methods need labeled rollouts, and both address how hard a problem is. Neither addresses when the model's own memory has become too noisy to benefit from more tokens. Koyuncu is the closest theoretical relative: it ties early-exit error exponents to the correlation between partial and full computation in the infinite-width limit. It does not study quantized caches or length dependence. Our quantity $\rho$ plays a similar role to its alignment measure, but it is measured on finite, real models.
Research Program
The first stage is measurement. For each layer and head of several open reasoning models spanning the 4B–32B range studied in Not All Bits Are Equal, and for 2-, 3-, 4-, and 8-bit KV quantizers, we estimate $\alpha$, $\sigma_e^2$, the correlation of $e$ with $k$ (which should be near zero by construction), and the empirical distribution of $\eta_j$. We also log attention margins $m$ on unlabeled reasoning traces.
The second stage validates the prediction on a controlled task. On synthetic key-retrieval prompts, we vary context length and bit width, and we compare predicted against observed retrieval-flip rates head by head.
The third stage builds the allocator. The gain correction rescales each dequantized key by $1/\alpha$, which costs one scalar per head. Bit allocation solves a small knapsack problem: it assigns bits to heads to maximize the minimum $\rho$ over retrieval-heavy heads under a memory cap. The per-query token cap tracks a running estimate of $n^\ast$ from the observed margins. When the current length approaches that estimate, the policy chooses between stopping and spending freed memory on parallel samples instead. Finally, we test whether the fitted $n^\ast$ curves, combined with model size, reproduce the precision-versus-length crossovers reported in Not All Bits Are Equal without fitting to accuracy.
Baselines
The naive reference point is the uniform deployment: a 16-bit KV cache where memory allows, or a single uniform bit width otherwise, with the same fixed token budget for every query. All methods are run on the same models, prompts, and seeds, at matched peak KV memory. We report accuracy at matched memory and memory at matched accuracy.
The strongest existing reference on this exact question is the best uniform configuration from the Not All Bits Are Equal grid. For each memory cap, we search precision and token budget over the same grid and pick the best on a held-out split, giving this baseline the same tuning budget our method gets. The paper reports only orderings, such as "32B model at 4-bit is strictly dominated by both a 14B at 8-bit and an 8B at 16-bit," and gives no numeric AIME accuracy; no reported result was found on Lacuna for per-benchmark accuracy of these configurations, so we reproduce the grid ourselves. Beating it means higher accuracy at equal memory on at least AIME and MATH 500, with a paired bootstrap interval that excludes zero.
For query-level stopping, the strongest competitor is thought calibration (Wu et al.), which reports "up to 60%" fewer thinking tokens on s1K-1.1 and a "20% reduction" on AIME and GPQA. We train its probe on the same quantized models. Our label-free cap counts as a success if it matches the probe's token savings at equal accuracy, or if it wins where the probe's calibration set is out of distribution.
Two ablations isolate the Bussgang ingredient. The first allocates bits per head by raw mean-squared quantization error, which removes the split between gain and noise. The second uses our allocator without gain correction.
Why not just…?
Why not just allocate bits by measured mean-squared error per head? This is the real threat. Once the gain is corrected, the remaining error is mostly the distortion term, so MSE-based ranking could perform about as well for bit allocation. The proposal's distinct content has two parts. One is the gain correction, which MSE ignores. The other is the length-dependent $\sqrt{2\ln n}$ prediction, which ties precision to token budget and which MSE ranking does not provide. The MSE ablation settles the first question. If MSE allocation plus gain correction matches our allocator, we will report that the benefit lies in the correction and the length law, not in the ranking.
Why not just train a probe to stop reasoning? Probes work: Zhu et al. report "ROC-AUC of 93.09%" for difficulty prediction on ScienceQA from the prompt's hidden state. Probes, however, need labeled rollouts for each model and each precision, and their guarantees weaken under distribution shift (Wu et al.). Nor do they separate "the problem is hard" from "the memory is too noisy." We run the probe as a baseline and also combine it with our cap, to test whether the two signals are complementary.
Why not just stop when sampled answers agree? Early-stopping self-consistency (Li et al.) needs no training and cuts samples by "80.1%" on GSM8K. It governs the number of parallel samples, not trace length or bit width. It also fails when the model repeats the same wrong answer, which quantization noise may make more likely. We include it in the parallel-sampling arm and measure whether consistent errors rise at low precision.
Why not just use infinite-width kernel theory? Koyuncu predicts early-exit exponents well ("~11.0" predicted versus "~11.75" measured on DistilBERT). That theory assumes infinite width, and the same paper notes that hard, sign-like non-linearities accumulate distortion quickly. Coarse quantizers are exactly this kind of non-linearity, and the Bussgang split handles them directly from finite-model statistics.
Proposed Experiments
We evaluate on AIME 24/25, MATH 500, GPQA Diamond, and LiveCodeBench, the suite used by Not All Bits Are Equal. Token budgets range from 2k to 30k and parallel sampling goes up to 16 paths, with KV precision from 2 to 16 bits.
The first decisive check is on synthetic retrieval. The per-head flip rate should follow the predicted $\sqrt{2\ln n}$ curve, and the fitted $\rho$ should rank heads by sensitivity with high rank correlation. If flips grow much faster than predicted, the uncorrelated-Gaussian model is wrong and the proposal fails at its core.
The second check is on real benchmarks. The plateau length of each accuracy-versus-budget curve should correlate with the model's retrieval-weighted $n^\ast$ across bit widths. A reasonable test is whether $n^\ast$ predicts the plateau within a factor of two.
The third check is the memory-matched comparison against the tuned uniform grid, the MSE ablation, and the no-gain-correction ablation. We also check whether the fitted curves recover the direction of the published crossovers without seeing any accuracy data.
References
- Linear One-Bit Precoding in Massive MIMO: Asymptotic SEP Analysis and Optimization
- Not All Bits Are Equal: Scale-Dependent Memory Optimization Strategies for Reasoning Models
- Throughput Analysis of Massive MIMO Uplink With Low-Resolution ADCs
- Quantized Precoding for Massive MU-MIMO
- LASSO with Non-linear Measurements is Equivalent to One With Linear Measurements
- On the Mean Square Error Optimal Estimator in One-Bit Quantized Systems
- Mitigating SAR-ADC Non-Idealities in Massive MU-MIMO Systems via Affine Models
- Learning How Hard to Think: Input-Adaptive Allocation of LM Computation
- Thought calibration: Efficient and confident test-time scaling
- Theoretical Guarantees for One-Shot Magnitude Pruning and Compute-Adaptive Early Exit
- The LLM Already Knows: Estimating LLM-Perceived Question Difficulty via Hidden Representations
- Escape Sky-high Cost: Early-stopping Self-Consistency for Multi-step Reasoning