When a language model answers from a retrieved document that contradicts what it learned in pre-training, it has to decide which source to trust. Models can fail in two ways. They can be stubborn and ignore correct new evidence, or they can be gullible and adopt wrong evidence. Interpretability work has located this decision in a small set of late-layer attention heads and mid-to-deep feed-forward blocks. Separately, low-bit quantization (storing weights in 2 to 4 bits instead of 16) is how most retrieval-augmented systems actually ship. All of that interpretability work was done on full-precision models. Quantization methods are judged on perplexity and multiple-choice accuracy, and those scores say nothing about whether a compressed model has quietly become more stubborn or more gullible.
This note makes two claims. First, low-bit quantization does not simply add noise to conflict behavior. It moves the model along the stubborn–gullible axis, and the direction depends on what the quantizer optimizes. Second, a quantizer can avoid that drift by allocating bits to preserve one scalar per conflict prompt: the full-precision model's log-odds of choosing the context answer over the memory answer. The research tests whether this targeted objective does better than the cheap fix, which is running an existing distribution-matching quantizer on conflict-heavy calibration text.
Why this gap is real, and what it is not
Two lines of work currently meet nowhere. Information-preserving quantization replaces raw rounding error with entropy or distribution criteria. Balanced Quantization sets bin boundaries at percentiles so every discrete level is used equally. IR-QLoRA searches a scale constant that maximizes the entropy of quantized base weights before low-rank fine-tuning. Bell Box Quantization rotates and maps weights to a uniform distribution during pre-training. SliM-LLM assigns more bits to groups along outlier activation channels, choosing the assignment that minimizes the divergence between quantized and full-precision output distributions on generic calibration text. These papers supply the methods this proposal reuses. None of them evaluates retrieval conflicts, and all of them rank parameters by statistical salience rather than by function.
On the conflict side, Cutting Off the Head Ends the Conflict uses path patching to find "memory heads" and "context heads" in late attention layers, then prunes them to steer reliance. ParamMute finds feed-forward blocks whose activation predicts overriding the context and mutes them. Analysing the Residual Stream shows with linear probes that conflict is detected around layer 14 and the winner is decided around layers 16–17. It also finds that context wins coincide with more concentrated, sparser activations, as measured by the Hoyer index. Competition of Mechanisms shows that factual recall partly works by suppressing the counterfactual token, and that two attention entries can swing GPT-2's recall from 4% to 50%. These papers motivate the proposal. They show that the decision runs through a few fragile components, exactly the kind of components that coarse rounding could perturb asymmetrically.
The research file reports no paper connecting the two directions. The obvious combination of the gaps in the file would be to deliberately quantize memory heads and the feed-forward blocks ParamMute flags to 1–2 bits so the model leans on context. I reject that as the core idea. It mixes compression with behavior editing, it would damage closed-book factual answering, and it would make the model more gullible to bad retrieval. That last risk matters because entity-centric conflicts are already dominated by persuasion: When Abundance Conceals Weakness reports an 81.0% persuasion rate against a 13.4% stubborn rate on PopQA. The goal here is narrower and more defensible. A compressed model should arbitrate the way its full-precision parent does. Steering methods such as PH3 or ParamMute can then be layered on top deliberately rather than introduced by accident.
Core Claim
Two hypotheses predict drift in opposite directions, and the study is built to tell them apart. The stubbornness hypothesis says that context heads depend on precise query–key matches over tokens the model has never seen, so they lose more under rounding than memory heads, which read out fixed associations. It adds that quantizers which spread out activation outliers would flatten the concentrated activations that Analysing the Residual Stream associates with context winning. The spreading happens through Hadamard rotation, the first step of Bell Box Quantization, or through equalization. The gullibility hypothesis says that entity-specific facts stored in feed-forward weights are fine-grained, and that low-bit rounding blurs them first, so the model defers to whatever the context says. Either outcome would be informative. A quantizer-dependent sign, where for example rounding-error methods drift one way and entropy-equalizing methods drift the other, would be the strongest and most useful result.
The proposed fix is an allocation objective. For a conflict prompt $x$ with context answer $a_c$ and memory answer $a_m$, define the arbitration margin $m(x) = \log p(a_c \mid x) - \log p(a_m \mid x)$. The search chooses a per-group bit assignment $b$ under an average budget $\bar{b}$:
$$\min_{b:\ \mathrm{avg}(b)\le \bar{b}} \ \mathbb{E}_{x\sim D_{\text{gen}}}\,\mathrm{KL}\big(p_{\text{fp}}(\cdot\mid x)\,\|\,p_{b}(\cdot\mid x)\big) \;+\; \lambda\,\mathbb{E}_{x\sim D_{\text{conf}}}\big(m_b(x)-m_{\text{fp}}(x)\big)^2$$
Here $D_{\text{gen}}$ is generic calibration text, $D_{\text{conf}}$ is a set of conflict prompts, the subscripts mark the full-precision and quantized models, and $\lambda$ trades the two terms. Full-vocabulary divergence on conflict prompts spreads its weight over every position and every plausible token. The margin term concentrates on the binary decision. That decision can move a long way while the divergence barely changes, because it depends on two tokens whose probabilities may both be small.
Research Program
The first stage is a measurement study. For Llama2-7B and Llama3-8B, the models with published conflict probes, and Qwen2.5-7B, which ParamMute studied, I quantize weights at 4, 3, and 2 bits with group size 128. Five quantizers are compared: round-to-nearest; GPTQ, an error-compensating rounding method; SliM-LLM; IR-QLoRA's entropy-maximizing scale calibration applied post-training without fine-tuning; and Hadamard-rotated activation quantization at 8 and 4 bits. The rotated condition uses only the rotation step and is not Bell Box Quantization itself, since its authors state that it is unsuitable for post-training use. For each condition I record conflict behavior along with three mechanistic readouts.
The first readout is the shift in path-patching importance of the previously identified memory and context heads. The second is the accuracy of the full-precision layer-16–17 arbitration probes when applied unchanged to the quantized residual stream. The third is the Hoyer index at layers 14–17, split by whether context or memory wins. Together these answer three questions: whether quantization moves arbitration, in which direction, and through which component.
The second stage builds the margin-preserving allocator. Evaluating every candidate allocation with forward passes is too costly. Instead, I rank groups by a first-order estimate of how much rounding them changes $m(x)$ on $D_{\text{conf}}$, combined with SliM-LLM's salience score, and then run SliM-LLM's double-pointer search with the combined objective. $D_{\text{conf}}$ is drawn only from the NQSwap training split. Every other conflict benchmark is held out, so the method cannot win by memorizing one dataset's entity distribution.
Baselines
The naive reference points are the full-precision model, which defines the target behavior, and GPTQ at the same bitwidth, which is what practitioners deploy by default. Beating GPTQ is necessary but not sufficient.
The strongest existing quantizer on the general criterion is SliM-LLM. It already matches output distributions instead of weight error, and it reports a 48% perplexity reduction over GPTQ for 2-bit LLaMA-7B and 81% over prior gradient-free methods on LLaMA-3-8B (SliM-LLM). I found no reported result on Lacuna for any quantized model on NQSwap, ConFiQA, CoFaithfulQA, or ConflictQA. Its conflict behavior must therefore be measured here, using the authors' group size and search procedure at identical average bitwidth.
The most dangerous competitor is conflict-calibrated SliM-LLM. This is unmodified SliM-LLM whose calibration set is the same NQSwap prompts, with retrieved passages, that my allocator sees. It receives the same data, the same number of calibration tokens, and the same search budget. The only difference is the margin term. It is the ablation that isolates the key ingredient.
A second ablation, circuit-labeled allocation, gives extra bits to the heads and feed-forward blocks that PH3 and ParamMute flag. It tests whether hand-labeled function is enough without the margin objective.
The entropy-maximizing quantizer from IR-QLoRA is included to test whether information preservation alone protects arbitration. That is a separate question from whether it improves accuracy, and the IR-QLoRA paper reports the accuracy gain (40.8% versus 38.4% MMLU at 4 bits) but has no conflict evaluation.
Beating a baseline means the following. At matched average bitwidth, including scale overhead, the method must reduce the absolute gap to full precision in context recall and memory recall on held-out CoFaithfulQA and NQSwap test data to at most half of conflict-calibrated SliM-LLM's gap. It must also keep the PopQA stubborn and persuasion rates closer to full precision. Finally, WikiText2 and C4 perplexity and closed-book NQ accuracy must stay within a tolerance fixed before the experiments are run.
Why not just…?
Why not just tell the model to rely on the context? Instructions move behavior in one fixed direction. The problem here is restoring a calibrated balance that compression disturbed. ContextFocus reports that instruction-only prompting leaves substantial unfaithfulness. A Causal View of Entity Bias finds prompting insufficient for high-frequency entities. Both findings come from full-precision models. I run an instructed condition for every quantizer anyway to see whether instructions recover the full-precision balance or just trade stubbornness for gullibility.
Why not just calibrate an existing quantizer on retrieval data? This is the real threat. Distribution matching on conflict prompts may already protect the components that make the decision, which would leave the margin term redundant. The conflict-calibrated SliM-LLM baseline exists to settle this question. If it closes the behavioral gap as well as my allocator does, I will report that the calibration data is the lever and the cheap fix suffices. That would still be a new and actionable finding, because no current quantizer is calibrated or evaluated this way.
Why not quantize normally and then fix behavior with steering or pruning? Steering vectors from ContextFocus and pruning from PH3 push the model toward context by a fixed amount on every prompt. They do not restore a prompt-dependent balance, and they would also make the model follow wrong context. The file also gives no evidence that a full-precision steering direction survives low-bit residual streams. I test this directly by applying ContextFocus vectors to each quantized model and measuring both faithfulness to correct context and resistance to misleading context, using the stubborn-rate split on ConflictQA. If post-hoc steering recovers both, the allocator is unnecessary at that bitwidth.
Why not just stay at 8 or 4 bits? If 4-bit models show no drift, this conclusion holds and the contribution narrows to 2–3 bits. The measurement study reports drift as a function of bitwidth precisely so that this can be decided from data.
Proposed Experiments
The decisive first check is a sign test. On NQSwap, CoFaithfulQA, and ConFiQA (scored with ps and po), I compare each quantizer against full precision and test whether the paired, per-example arbitration margin $m(x)$ shifts systematically. COUNTERFACT serves as a mechanistic control. The full-precision anchors in the literature are linear-probe conflict detection near 90% at layer 14 and arbitration AUROC above 0.85 at layers 16–17 in Llama3-8B (Analysing the Residual Stream). Probe degradation on quantized activations tells us whether the decision signal itself was damaged.
The second check is attribution. If the shift tracks loss of context-head importance or a drop in context-winning Hoyer concentration, the stubbornness mechanism is supported. If it tracks a loss of closed-book recall on the same entities, the gullibility mechanism is supported. Rotation-based activation quantization gives a direct test of the concentration story, because it should lower the Hoyer index and push the model toward stubbornness if the hypothesis is right.
The third check is the allocator comparison under the Baselines criteria on held-out benchmarks. ConflictQA PopQA and StrategyQA stubborn and persuasion rates, with full-precision references of 13.4% and 81.0% on PopQA and a 28.4% stubborn rate on StrategyQA (When Abundance Conceals Weakness), measure whether gullibility was traded for stubbornness. Perplexity and closed-book NQ guard against the method buying conflict fidelity with general capability.
The idea is falsified in two cases: if conflict-calibrated SliM-LLM matches the allocator at every bitwidth, or if no quantizer shifts the margin beyond seed noise at 3 bits and above.
References
- Balanced Quantization: An Effective and Efficient Approach to Quantized Neural Networks
- Accurate LoRA-Finetuning Quantization of LLMs via Information Retention
- Boosting Entropy with Bell Box Quantization
- SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models
- Cutting Off the Head Ends the Conflict: A Mechanism for Interpreting and Mitigating Knowledge Conflicts in Language Models
- ParamMute: Suppressing Knowledge-Critical FFNs for Faithful Retrieval-Augmented Generation
- Analysing the Residual Stream of Language Models Under Knowledge Conflicts
- Competition of Mechanisms: Tracing How Language Models Handle Facts and Counterfactuals
- When Abundance Conceals Weakness: Knowledge Conflict in Multilingual Models
- ContextFocus: Activation Steering for Contextual Faithfulness in Large Language Models
- A Causal View of Entity Bias in (Large) Language Models