Does Low-Bit Quantization Make Language Models Stubborn?

When a retrieval system hands a language model a document that contradicts what the model memorized during training, the model has to decide whom to believe. Interpretability work has shown that this decision is made by a small, identifiable set of internal components in the middle and late layers. That same work has shown that a model choosing to follow the document produces unusually concentrated, skewed internal activations. Quantization, meanwhile, rounds activations onto a coarse grid so models run cheaply. Its standard tools treat skewed, heavy-tailed activations as a nuisance and flatten them. Nobody has checked whether compressing a model quietly changes how it resolves these conflicts, for example by making it ignore retrieved evidence while its perplexity and benchmark scores look fine.

This note proposes to measure that effect and locate it inside the network. It then tests one specific, cheap repair: give the activations at the few sites where the conflict is decided a quantization grid with its own offset from zero (asymmetric quantization), calibrated on conflict prompts. The deliverable has two parts. The first is a metric and a map: a per-scheme, per-site account of whether quantization pushes a model toward its memory or toward its context. The second is a test of whether the repair beats the obvious alternatives at equal cost. The two-sentence version is this: we suspect that activation quantization biases compressed models toward their pretrained beliefs and against retrieved evidence, in a way standard quantization benchmarks cannot see. We will measure that bias, find where it comes from, and check whether fixing the grid at those few places removes it.

Core Claim

The claim is that activation quantization causes a directional drift in knowledge arbitration, beyond any uniform loss of accuracy. On conflict benchmarks, the drift should show up as a rise in the fraction of answers that come from memory rather than context. This rise should be larger than any drop on matched prompts whose context agrees with memory. Our second claim is that the drift is concentrated at the arbitration sites already identified in full precision. Its size should depend on how the activation grid is placed (symmetric vs. offset, and the basis set by rotation), not only on bit width.

Three lines of prior work motivate this. Analysing the Residual Stream of Language Models Under Knowledge Conflicts linearly probes LLaMA-2-7B and LLaMA-3-8B. It finds that conflict detection peaks around layer 14 (about 90% probe accuracy) and the model's eventual choice becomes predictable at layers 16–17 (AUROC above 0.85). It also finds that following context goes with higher kurtosis and a higher Hoyer index (a sparsity measure). Cutting Off the Head Ends the Conflict isolates competing "memory heads" and "context heads" in late attention layers. ParamMute identifies mid-to-deep feed-forward blocks that inject memorized facts over context, and reports that larger models are more stubborn. All three work only in 16-bit precision. We reuse their localizations as the candidate sites, and we do not propose a new conflict-resolution method of our own.

On the quantization side, SpinQuant learns rotations that drive activation kurtosis from over 200 to about 3 before 4-bit quantization. AMXFP4 shows that small quantization groups become asymmetric and that rotation worsens this. Evaluating Quantized Large Language Models finds larger models disproportionately hurt by activation quantization. That study used symmetric per-token activation grids and tested hallucination and retrieval only in conflict-free settings. None of these papers evaluates a single conflict benchmark.

We have to address one apparent contradiction directly. A rotation is lossless in full precision, so "Gaussianizing" activations cannot by itself destroy the context signal. The Hoyer-index finding is measured in the original basis. Our mechanistic hypothesis is therefore narrower. The context-override signal may be a small-norm direction riding next to large outlier channels. When the per-token scale is set by those outliers, rounding error can swamp that direction. Rotation and grid placement each change how much of that small signal survives rounding. Whether they help or hurt is an empirical question, and we treat it as one rather than assuming the skew-kills-context story.

Research Program

The first stage is measurement. Following ContextFocus, we score each conflict item by whether the model produces the context (substituted) answer, $p_s$, or the memorized (original) answer, $p_o$. We summarize these as the memorization ratio $MR = p_o/(p_o+p_s)$. Arbitration drift is the change in $MR$ from the 16-bit model to the quantized one. To separate drift from generic damage, every conflict item is paired with a control: the same question with an agreeing context, plus a closed-book version. A scheme that merely makes the model worse should lower both conflict and control accuracy. A scheme that makes it stubborn should raise $MR$ while control accuracy stays near full precision.

The second stage is localization. Starting from a fully quantized model, we restore one candidate site at a time to 16 bits. Conversely, starting from the 16-bit model, we quantize one site at a time. The candidate sites are the layer 8–17 residual stream, the memory and context heads found by path patching, and the identified feed-forward blocks. We also retrain the layer-14 detection probe and the layer-16/17 decision probe on quantized activations. This tells us whether quantization erases the model's awareness of the conflict, or only its follow-through on that awareness.

The third stage is the repair. At the localized sites only, we replace the symmetric activation grid with a learnable-offset grid in the style of LSQ+, where a zero-point and step size are fitted by mean-squared-error initialization and folded into biases where possible. These grids are calibrated on a mixture of the standard corpus and roughly 1.5k conflict prompts, matching the data budget ContextFocus used for its steering vectors. The rest of the model keeps its original quantizer. Restricting asymmetry to a few sites also limits the zero-point bookkeeping overhead that commodity integer hardware handles poorly, as AMXFP4 notes.

Baselines

The naive reference points are the 16-bit model, which defines zero drift, and round-to-nearest symmetric per-token W4A4 activation quantization, the scheme used in Evaluating Quantized Large Language Models.

The strongest existing low-bit method is SpinQuant at W4A4KV4. On LLaMA-2 7B it reports an average zero-shot accuracy of 64.0, a 2.9-point gap to FP16 (SpinQuant). We will run its released procedure with its default calibration, and again with our conflict-augmented calibration set, at the same rotation-learning budget. AMXFP4 is the strongest asymmetric format. It reports WikiText-2 perplexity of 6.22 on LLaMA2-7B, against 11.15 for MXFP4 and 8.28 for rotation-based INT4 (AMXFP4). We include it, simulated in software, to test whether asymmetry applied everywhere already removes the drift. For conflict metrics on any quantized model, no reported result was found on Lacuna. For full-precision conflict methods, ParamMute reports a 5% average gain in context recall on CoFaithfulQA, and PH3 reports a 6.2% recall gain on NQ. These anchor what size of $MR$ shift counts as practically meaningful.

The key ablations isolate the ingredient. The first gives the same sites extra bits under symmetric grids at an equal total bit budget. The second uses asymmetric grids at randomly chosen sites. The third uses asymmetric grids at the localized sites but calibrates them on standard data only. The repair counts as a success under three conditions. It must reduce drift in $MR$ by at least half relative to SpinQuant on NQ-SWAP and ConFiQA, measured against full precision. It must beat the equal-budget mixed-precision ablation. And it must stay within 0.5 points of SpinQuant's zero-shot average and WikiText-2 perplexity.

Why not just…?

Why not just calibrate the existing quantizer on retrieval-style conflict data? This is the most serious threat. The whole drift could be a calibration mismatch: standard quantizers fit their ranges on WikiText or C4, which contain no conflicts. We therefore run SpinQuant with our exact conflict-augmented calibration set. If that alone closes the drift, the honest conclusion is that calibration data, not grid shape, is the lever. The paper would then report the measurement and localization results with that simpler fix, and the asymmetric grid would be dropped from the claim.

Why not just keep the arbitration layers in higher precision? Mixed-precision allocation, as in TWLA's inter-layer allocation (TWLA), is a natural competitor, and it may well work. Its allocation is driven by general interaction loss and knows nothing about arbitration circuits. Our equal-budget ablation compares bits versus offsets at the same sites. If extra bits win, the contribution becomes a conflict-aware bit-allocation criterion, which is still enabled by the localization stage.

Why not just apply activation steering or head pruning after quantizing? ContextFocus and PH3 could in principle undo the drift. However, they add a fixed vector or zero out heads on activations that are then rounded to 16 or fewer levels. A steering vector added at multiplier 2 can push values into clipped bins that calibration never saw. We test this directly by applying ContextFocus to each quantized model, and we report whether its effect on $p_s$ shrinks relative to full precision. If steering fully survives, it becomes a cheap complementary fix rather than a replacement for diagnosis.

Why not just use weight-only quantization and leave activations in 16 bits? Weight-only 4-bit is near-lossless in Evaluating Quantized Large Language Models and serves as a useful control. If it shows no drift, that confirms the effect is specific to activations. It does not remove the need for activation quantization, which is where throughput gains come from.

Proposed Experiments

We will run LLaMA-2-7B and LLaMA-3-8B, matching both the residual-stream study and SpinQuant, plus a 13B model to test whether ParamMute's larger-models-are-stubborner finding compounds with the activation-quantization sensitivity of large models. The conflict benchmarks are NQ-SWAP, ConFiQA (single-hop, multi-hop, and multi-counterfactual), CoFaithfulQA, Macnoise and ConflictQA. General quality is tracked with WikiText-2 perplexity and the zero-shot suite. The schemes are W4A16, W4A8, and W4A4 for round-to-nearest, SpinQuant, simulated AMXFP4, and the repair, with three calibration seeds each.

The first decisive check is whether drift exists at all. At W4A4, $MR$ should rise more than control accuracy falls, with confidence intervals excluding zero. If it does not, the core claim is false and we report a clean negative result, which would itself be useful to deployers of retrieval systems. The second check is whether drift is local. Restoring the layer 14–17 sites should recover most of the $MR$ shift, while restoring an equal number of random layers should not. The third check is whether the probes tell detection from decision. If the layer-14 detection probe survives quantization but the decision probe degrades, the damage is in follow-through, and that predicts the repair should target layers 16–17 and the context heads. The fourth check pits the repair against conflict-calibrated SpinQuant and equal-budget mixed precision, using the success thresholds stated in the Baselines section.