Repair-on-Escalation: Precision Switching That Fixes the Past, Not Just the Future

A heavily compressed language model often starts an answer well and then derails dozens of tokens later. It repeats itself, loses the thread of a multi-step argument, or starts hallucinating. Storing weights in 2 or 3 bits instead of 16 makes the model small and fast, but it sits right at the edge of a known "accuracy cliff." A natural remedy is to watch the model while it generates and switch it to higher precision when it starts to fail. Weight formats now exist that make such switching cheap: in binary-coded "bit-plane" formats, a 4-bit model is literally the 2-bit model plus extra residual planes of weights. The catch is that switching is a form of reconfiguration. Like any reconfiguration, it pays off only when the time it saves exceeds what it costs.

This note argues that the real cost of switching precision mid-generation has been misidentified. Prior work blames kernel launch and host-side overhead. We argue the larger cost is stale state. Every token generated at low precision has already written its attention keys and values (the per-token memory the model rereads at every later step, the "KV cache") using degraded weights. Moving to higher precision from now on leaves that corrupted memory in place. We propose that escalation should roll back a short window of recent tokens and recompute them at high precision in a single parallel pass. We then derive when doing so beats simply staying at high precision.

Core Claim

On a nested bit-plane model, a closed-loop controller does four things. It detects impending failure from internal signals, rolls back and re-encodes the last $w$ tokens at higher precision, holds that precision for a period, and then drops back. We claim this recovers most of the quality gap to a uniform high-precision model at lower average latency than three alternatives: uniform middle precision, the progressive schedules of PMPD, and forward-only escalation. We also claim that the rollback step is what makes escalation work at all. Forward-only escalation, in this view, mostly fails to rescue generations whose KV cache has already drifted.

The claim is falsifiable in two ways. Either repair adds nothing over forward-only switching, or unconditional periodic verification (the speculative-decoding special case, discussed below) matches it.

The breakeven condition extends the one formalized in Efficient Interfacing of Partially Reconfigurable Instruction Set Extensions, which states that reconfiguration helps only when time saved exceeds reconfiguration latency. To add state repair, we use the following symbols. Let $t_{hi}$ and $t_{lo}$ be per-token decode times at high and low precision. Let $N_{lo}$ be the number of tokens decoded at low precision. Let $E$ be the number of escalations in a generation, $t_{sw}$ the fixed switch cost, and $t_{rep}(w)$ the time to re-prefill $w$ tokens at high precision. The controller beats always-high decoding when

$$N_{lo}\,(t_{hi}-t_{lo}) \;>\; E\,\big(t_{sw} + t_{rep}(w)\big).$$

Two properties of this inequality matter. First, re-prefill is compute-bound and parallel over $w$ tokens, whereas decode is memory-bound, so $t_{rep}(w)$ should grow far more slowly than $w\,t_{hi}$. This is the same reason prompt processing is cheap per token. Second, the controller can tune $E$ through its trigger threshold and hysteresis. The research question is whether realistic detectors keep $E$ small enough.

Research Program

We build on three pieces of prior work, each in a different role.

AnyBCQ supplies the mechanism. It represents weights as sums of binary planes, freezes a 2-bit base, and adds residual planes for 3 and 4 bits. Its kernels avoid the lookup overhead, which it measured at 35–58% of kernel latency for clustering-based multi-precision formats. Changing precision therefore means reading more planes, not loading a different model. AnyBCQ provides no runtime policy for when to switch.

Cognitive Fatigue in Autoregressive Transformers supplies the detector. Its Fatigue Index combines three signals: decaying attention to the prompt, drift in the model's internal representations, and deviation in output entropy. It predicts task degradation with AUROC 0.95. Its hysteresis alerting, which requires a signal to persist before firing and to clear before resetting, cuts alert jitter by over 91%. That paper also reports that 4-bit quantization makes models overconfident and repetitive earlier. It proposes no intervention.

Progressive Mixed-Precision Decoding (PMPD) both motivates and constrains the design. It shows that precision sensitivity varies over a generation: prefill and early tokens are sensitive, later tokens are resilient. It lowers precision on a schedule that is either static or predicted once from the KV cache. It also reports that host-side kernel reconfiguration becomes a bottleneck. Our controller runs in the opposite direction, escalating upward and reactively. It also treats state repair as a first-class cost, which PMPD does not.

We will build four things.

The first is the controller. It computes a lightweight probe every token. We will test the Fatigue Index, the top-1 versus top-2 logit margin (the confidence test Cascade^CNN used for images), and a small learned probe on the final hidden state. When a probe fires, the controller discards the KV entries for the last $w$ tokens and re-prefills those tokens at high precision. By default it keeps the tokens themselves and replaces only their cache. A "resample" variant regenerates them as well.

The second is a set of low-overhead switching kernels. We pre-capture one GPU execution graph per plane count, so that $t_{sw}$ approaches a kernel dispatch. This deliberately removes the host-side bottleneck PMPD reported, so that any remaining gap can be attributed to state rather than to launch overhead.

The third is a measured breakeven model. We will profile $t_{hi}$, $t_{lo}$, $t_{sw}$, and $t_{rep}(w)$ on RTX 4090 and A40 GPUs, the hardware PMPD used, and in the FlightLLM NPU simulator. We then check whether the inequality above predicts the measured end-to-end speedup across precision pairs.

The fourth is a secondary analysis for reconfigurable hardware. On an FPGA whose DSP chains accumulate bit-planes in the style of FlightLLM, adding a plane means enabling extra accumulations, not swapping a bitstream. That avoids the 38–42 ms partial-reconfiguration stall measured in AH-CNN. We will estimate this cost analytically and in simulation only. We make no claim of a silicon build.

One honest risk shapes the plan. AnyBCQ's 2-bit base is weak: its LLaMA-3.1-8B MMLU score is 35.32. A 2→4 bit controller may therefore escalate so often that it fails breakeven. We will sweep base/high pairs of 2→3, 2→4, and 3→4, and expect 3→4 or 2→3 to be the useful operating points.

Baselines

The naive reference points are uniform decoding at the high precision and uniform decoding at the middle precision, using the same AnyBCQ bit-plane weights and kernels. The high-precision run sets the quality ceiling and the latency to beat. The middle-precision run is the "just pick 3 bits" answer. Any controller whose average plane count exceeds 3 must beat it on quality, and any controller below 3 must match it. Evaluating Quantized Large Language Models reports 4 bits as near-lossless, so the realistic contest is between our method and uniform 3-bit, not uniform 4-bit.

The strongest existing method is PMPD. We will run both of its schedulers, the static one calibrated on C4 and the learned one that reads the KV cache, on the same bit-plane weights and the same pre-captured kernels. This gives PMPD our switching-cost advantage and isolates the policy difference. Both methods get the same calibration data and the same budget for tuning thresholds. PMPD reports lossless CNN/DM and Dialogsum at up to 33% lower average bit-width. It reports 13.9–18.4% higher Rouge-L than Dense-and-Sparse baselines on MT-Bench at equal average bit-width, and up to 1.41x speedup over uniform quantization. Beating PMPD means one of two things: higher task score at equal measured wall-clock latency, or equal score at lower latency, on the same tasks. Average bit-width is not the comparison metric, because it hides switch and repair costs.

The ablation that isolates the key ingredient is forward-only escalation. It uses the same probe, thresholds, and hysteresis, but performs no rollback, so stale KV entries stay in place. A second ablation fires escalations at random with the same frequency and duration. This tests whether the probe carries information or whether simply spending more high-precision tokens explains the gain.

The final baseline is nested bit-plane speculative decoding. The low-plane model drafts $k$ tokens, and the full-plane model verifies them in one parallel pass. This is the unconditional, periodic form of our repair step. No reported result found on Lacuna for speculative decoding with nested bit-plane drafts, so we will implement it with $k$ tuned on the same budget as our controller's $w$.

Why not just…?

Why not just use static mixed precision, keeping sensitive layers high forever? Static allocation, such as the spectral bit assignment of AlphaQ, removes switching entirely. It cannot respond to failures that begin mid-generation, which Cognitive Fatigue documents. PMPD also shows that the same layer's sensitivity changes over decoding. That said, a good static 3-bit allocation is a real threat at short outputs. We expect our controller's advantage to appear mainly on long generations, such as multi-hop HotpotQA and dialogue, and we will report results stratified by output length so a null result at short lengths is visible.

Why not just remove the host overhead with pre-captured kernels and use PMPD as is? We do exactly this for PMPD in our comparison, so the question becomes empirical. If PMPD with cheap switching matches our controller, then the reconfiguration problem reduces to engineering and our state-repair hypothesis is wrong. The forward-only ablation settles the mechanism independently of the policy.

Why not just do speculative decoding with the low-plane model as drafter? This is the most serious alternative. It guarantees output identical to the high-precision model and amortizes the plane switch over $k$ drafted tokens, just as batching amortizes reconfiguration in Cascade^CNN. Its weakness is acceptance rate. A drafter at the accuracy cliff will have many draft tokens rejected, and every window pays for verification whether or not anything went wrong. Our method pays for repair only when the probe fires. If speculative decoding matches us at equal latency, we will report that periodic verification dominates triggered repair, and that the probe's value lies only in choosing $k$ adaptively.

Why not just keep separate 2-bit and 4-bit models and route whole queries? AnyBCQ reports that separate models cost 49% more memory than shared planes. Routing at the query boundary also cannot catch a generation that fails partway through.

Proposed Experiments

We will evaluate LLaMA-3.1-8B, which gives a direct comparison to AnyBCQ's MMLU numbers, and Vicuna-7B and MobileLLaMA-1.4B, for comparison to PMPD's simulator results of over 15 and over 50 tokens/s. Tasks are CNN/DM, Dialogsum, and MT-Bench from PMPD, and long-form HotpotQA, TriviaQA, and SQuAD generation following the Cognitive Fatigue protocol. We also report WikiText-2 perplexity and the six common-sense reasoning suites as sanity checks.

Metrics are task score (Rouge-L, exact match, accuracy) and measured time per token at the median and 99th percentile, at batch size 1 and small batches. We also report escalation rate, the rate of repetition loops, and the fraction of generations that fail and are recovered.

The decisive checks are three. First, at matched wall-clock latency, the rollback controller must beat forward-only escalation on long generations, with a larger gap when $w$ covers the tokens decoded before the probe fired. Otherwise the stale-state claim fails. Second, the probe-driven controller must beat random triggering at the same escalation rate. Third, the breakeven inequality, using measured $t_{rep}(w)$ and $E$, must correctly predict for which precision pairs and output lengths the controller outperforms uniform high precision. The controller must then either beat, or clearly lose to, speculative decoding. We report whichever outcome occurs.