Precision at the Cut: Rescoring Only the Borderline Hypotheses in Quantized Search

When a language model is compressed to very few bits per number, its predictions get slightly noisy. For greedy decoding, where the model always takes its single top choice, a little noise matters only when it flips that one choice. Search-based decoding is different. Beam search keeps several partial answers at once, and thought-level tournaments rank whole reasoning branches. Both repeatedly cut a ranked list of partial answers at a fixed size, and anything below the cut is gone for good. This note proposes that much of the reasoning loss seen in quantized models under search is premature pruning caused by quantization noise at the cut. A correct partial answer that the full-precision model would have kept sits just below the threshold under the noisy model and is discarded. Nothing later in the search can bring it back.

The proposed fix is cheap and targeted. Run the search with the fast low-precision model. At each cut, re-score with a higher-precision mode of the same weights only the few candidates whose low-precision scores fall within a calibrated noise band around the threshold. Everything far above or far below the line is decided by the cheap model. Only the borderline cases pay for precision, and their recomputed high-precision memory states replace the noisy ones for the survivors. If quantization errors are small relative to typical score gaps, most cuts need little or no rescoring, and the search should recover most of the accuracy of the high-precision model at close to low-precision speed.

Core Claim

The claim has two parts, and both can be tested. The first is diagnostic. Under 4-bit activations (W4A4) and 2-bit weights, the drop in search-decoded accuracy on reasoning benchmarks is explained to a large degree by membership flips at the beam cut: candidates that the high-precision model ranks inside the beam but the quantized model ranks just outside. These flips should concentrate among candidates whose quantized score is close to the cut. The second is constructive. Rescoring only a margin band of width $\delta$ around the cut with a higher-precision mode recovers most of that loss at a small fraction of the cost of running high precision everywhere.

Concretely, suppose at one pruning step there are $C$ candidates with low-precision scores $s_q$, and we keep $B$ of them. Let $\tau$ be the $B$-th largest $s_q$. The rescoring band is

$$\mathcal{R} = \{\, c : |s_q(c) - \tau| < \delta \,\},$$

where $\delta$ is set to a high quantile (for example the 95th) of $|s_q - s_h|$. Here $s_h$ is the high-precision score, measured on a small calibration set. Candidates in $\mathcal{R}$ are re-ranked by $s_h$. All others keep their $s_q$ ranking. The quantile rule follows directly from the definition: if a candidate's error is below $\delta$, it can only be misplaced relative to the cut if it lies inside the band. So the band catches the corresponding fraction of flips, apart from flips caused by noise in $\tau$ itself.

Relation to Prior Work

Three papers motivate the gap, and none of them closes it. Leveraging Cross-Utterance Context For ASR Decoding shows in speech recognition that rescoring after the search underperforms putting the stronger model inside the search. On AMI, beam search with shallow fusion reduced word error rate by 2.4%–2.5% absolute over greedy decoding, against 2.0%–2.1% for N-best rescoring, because correct transcripts had already been pruned. Factorization of Language Constraints in Speech Recognition and Multi-Task Language Modeling for Improving Speech Recognition of Rare Words make the same structural point: a second pass can only choose among first-pass survivors. Our proposal applies this lesson to precision rather than context. The "weak first pass" is the quantized model, and the stronger scorer is invoked at the cut rather than after it.

On the quantization side, An empirical study of LLaMA3 quantization reports that sub-3-bit models keep their fluency but collapse on GSM8K, MATH and HumanEval. Evaluating Quantized Large Language Models finds activation quantization especially fragile in large models. Both measure greedy or perplexity behavior, not survival of the correct path inside a beam. That missing measurement is our first contribution.

Two works supply method. QSpec runs one weight-quantized model in two modes, a fast W4A4 drafter and a W4A16 verifier. It overwrites the drafter's attention key/value memory (the "KV cache") with the verifier's "golden" entries. This restores MATH accuracy from about 12–15% back to the roughly 24% of W4A16, at 1.32×–1.64× throughput. We reuse its dual-mode model and its KV overwriting. QSpec, however, verifies a single greedy draft token by token, so it has no notion of a beam cut and cannot keep alternatives alive. Thought-Level Beam Search for Reasoning (Gambit) runs periodic tournaments over reasoning branches, scored by a small probe on hidden states. It gains +6.7% on HMMT-24 and +3.3% on AIME-25 over the strongest pruning baselines. It was evaluated only at full precision. We use its tournament as the second search setting, and we ask whether the probe's scores survive quantization.

Two works constrain the design. HierSpec shows that verifying large token trees with 4-bit kernels incurs a "verification tax" that cancels the speedup. Our rescoring therefore has to be small and batched: a handful of sequential continuations per cut, never a tree. HierSpec also relies on a separate FP16 model to make pruning decisions; we avoid any second model. AnchorKV makes the analogous argument for memory: permanently evicting tokens is a form of premature pruning (Needle-in-a-Haystack drops to 0.18 under eviction versus 0.94 when tokens are kept in compressed form). AnchorKV acts on the memory of a fixed sequence, not on which hypotheses a search keeps.

For the 2-bit regime we need a high-precision mode that shares storage with the low-precision one. We take inspiration from the additive extended codebook of CRVQ and the two-stage scheme of GPTQT. The plan is to quantize weights to 2 bits with QuIP, whose incoherence processing gives usable 2-bit perplexity (6.33 on Llama 2 70B), and then store a separately quantized 2-bit residual. The fast mode reads only the base weights; the precise mode adds the residual.

Research Program

The work proceeds in three stages. Each stage can stop the project if its premise fails.

Stage 1: survival analysis. We take Llama-family models at FP16, W4A16, W4A4 and 2-bit QuIP. We run token-level beam search on GSM8K and MATH-500, and Gambit-style thought tournaments on AIME and HMMT. At every cut we record:

  • the cut threshold;
  • every candidate's low-precision and high-precision scores;
  • whether a prefix of a correct final solution was present and survived.

For token-level beam search, "correct prefix" means a prefix of a trace that the FP16 search eventually completes correctly. This yields three quantities: the survival curve of correct prefixes under each precision, the rate of membership flips, and how the flips are distributed against margin to the cut.

Stage 2: margin-band rescoring. We implement the band rule with $\delta$ calibrated on held-out problems, together with KV overwriting for survivors. For Gambit we apply the same idea to probe scores. We also test whether the probe itself must run in high precision or can be retrained on quantized hidden states.

Stage 3: cost accounting. We measure the fraction of candidates rescored, the number of high-precision forward tokens, wall-clock throughput with real W4A4/W4A16 kernels, and the resulting accuracy–throughput frontier.

Baselines

Naive reference. The quantized model with the same search procedure and no rescoring, alongside greedy decoding at each precision. This shows how much search alone helps or hurts a quantized model. We found no reported result on Lacuna for beam search or thought-level search on quantized models, so these numbers must be produced here.

Upper reference. The same search run entirely in W4A16, or FP16 for the 2-bit experiments. This defines the accuracy we are trying to recover.

Strongest existing method: QSpec. QSpec is run with its released configuration on the same models and MATH/GSM8K splits. Its reported number is MATH accuracy restored from about 12–15% (W4A4) to about 24% (W4A16 level), at 1.32×–1.64× throughput over W4A16 (QSpec). Because QSpec is greedy, we additionally build "QSpec-beam": QSpec verification applied to every beam continuation. This is the obvious combination, and it tells us whether targeting the band matters or whether verifying everything is just as good. Beating QSpec means higher accuracy at equal or better throughput, or equal accuracy at clearly higher throughput.

Gambit at full precision. On the thought-level setting, the reference is Gambit's reported gains (+6.7% HMMT-24, +3.3% AIME-25 over pruning baselines, Gambit). We reproduce it at FP16 and at each precision.

Fairness. All methods share the same models, data splits, prompts and decoding length limits. Any method with tunable parameters (beam width, $\delta$, number of rescored candidates) gets the same tuning budget on a common validation split. Comparisons are reported at matched wall-clock and matched high-precision token counts.

Key ablation. Rescoring a random subset of candidates, with the same size as the band, at each cut. This isolates whether the gain comes from targeting the margin or merely from spending some high-precision compute. A second ablation drops the KV overwriting to separate better pruning decisions from cleaner memory.

Why not just…?

Why not just widen the beam of the quantized model? This is the most serious threat. If flips sit just below the cut, a slightly wider beam would keep them too. The difference is what the extra compute buys. Widening keeps more low-precision candidates and lets them grow for many steps, so the cost compounds. It also does nothing about candidates misranked far from the line. HierSpec's verification tax suggests that wider candidate sets are expensive with quantized kernels. We will settle this empirically: give the quantized model the widest beam that fits the same wall-clock budget as band rescoring, and compare accuracy. If wider beams match us, the honest conclusion is that quantization damage under search is mostly about beam width, and the proposal reduces to choosing beam width under quantization.

Why not just rescore the final N-best list with the high-precision model? Second passes are limited to whatever survived the first pass. The AMI result in Leveraging Cross-Utterance Context For ASR Decoding shows the cost of this in speech: 2.0%–2.1% versus 2.4%–2.5% absolute word error reduction. Our Stage 1 survival curves will measure directly how often the correct prefix is already gone by the end. Post-hoc rescoring at matched high-precision cost is included as a baseline.

Why not just run W4A16 throughout, or use a better quantizer? If the fast mode is not meaningfully faster, there is nothing to gain. QSpec reports that it is, at 1.32×–1.64×. A better quantizer such as QuIP or CherryQ shrinks the noise but does not remove the cut, and we apply band rescoring on top of the best available quantizer rather than instead of it. The method loses its point if the band must cover most candidates. We report the rescored fraction for exactly this reason.

Decisive Checks

The project hinges on one measurement from Stage 1. If membership flips are not concentrated near the cut, the band rule has no target. This would happen at 2 bits if distributions collapse into the repetition and gibberish that An empirical study of LLaMA3 quantization describes. We then expect the proposal to work at W4A4 and fail at 2 bits, and we would report that boundary as the finding.

Success at W4A4. On MATH-500 with beam search, band rescoring should reach at least 90% of the W4A16 search accuracy while rescoring under 20% of candidates. It must also beat both the random-subset ablation and the matched-latency wider beam.

Thought-level setting. The test is whether Gambit's tournament gains persist when the probe reads quantized hidden states. If they vanish and are restored only by high-precision probe scores at the cut, that directly answers the open question of whether quantization corrupts the value estimates that search relies on.