Hospitals that want to run language models on their own hardware usually compress them first. Post-training quantization rounds a trained model's weights to 4, 3 or even 2 bits so it fits on a local server. The accepted check is whether benchmark accuracy survives. Our worry is narrower and, in clinical use, more dangerous. A compressed model may keep producing plausible answers while it quietly stops reading the patient record and falls back on what it memorized during pretraining.
If that happens, the explanation a clinician audits is tied to a generic patient, not this one, and accuracy on standard benchmarks would not reveal it. We propose to measure this failure directly. The test edits a single value in a patient note (a creatinine level, an age, a heart rate) and asks whether the compressed model's answer and rationale change the way the edit requires. We then build a quantization method that decides which weights to keep at high precision by how much they contribute to this record-dependence, rather than by how much they contribute to task accuracy.
Core Claim
We make three claims. First, weight quantization at 4 bits and below erodes record-dependence before it erodes accuracy. Second, the erosion can be localized to a small set of weights. Third, protecting those weights preserves grounded clinical reasoning better than protecting weights chosen for task loss, at the same bit budget.
The new ingredient is the objective that drives the choice of protected weights. It is a counterfactual record-sensitivity signal. Neither the task loss used by circuit-guided quantization nor generic calibration text provides it.
What Prior Work Establishes and Leaves Open
Several studies motivate the concern without testing it.
- Does Accuracy Equal Evidence? shows that compressing the decoding-time key-value cache preserves answers while severing the reasoning chain. On AIME, accuracy stayed at 73% of the original while consistency fell to 39%, with a Spearman correlation of −0.95 between the two. That study covers only cache compression, not weights, and is purely behavioral.
- Can Large Language Models Still Explain Themselves? finds that weight quantization lowers human-rated explanation coherence by up to 8.5%, while automatic metrics barely move. Its only clinical data are short health claims (HealthFC), where there is no record to ground in.
- Through a Compressed Lens localizes quantization damage to the final layers and to the first hop of multi-hop recall. It also reports a collapse in 4-bit GPTQ factual recall on Qwen2.5-14B, from 73.08% to 25.20%. All of this uses open-domain fact triples.
Together these suggest that compression attacks the pathway from input evidence to output, but none tests a model's dependence on a patient record.
Two papers supply our evaluation tools.
- The Hallucination Dependence Index compares unsupported-claim rates with and without the patient bundle on 119 TCGA-LUAD cases, and shows that grounded metrics hide large differences in prior reliance. Its HDI ranged from 0.336 to 0.984 across models. However, it covers only full-precision API models.
- MedCalc-Bench contains 1,047 notes with calculator ground truth. It attributes roughly half of all failures to extracting the wrong variable from the note. That is exactly the record-reading step we care about, and it lets us edit a note and recompute the correct answer from the formula.
One paper supplies the method we modify. Task-Circuit Quantization (TACQ) keeps the top-p% of weights in 16-bit, ranked by a saliency score. The score combines simulated-quantization loss change with magnitude-sharpened gradients. TACQ recovers 96% of MMLU at 3.1 bits on Llama-3-8B-Instruct. Its saliency is computed from task loss, so it protects whatever produces correct answers, including memorized shortcuts.
Two findings constrain our design.
- Quantization Hurts Reasoning? places W8A8 and W4A16 in a "lossless" zone of at most a 1% drop. It also reports that domain-matched calibration data improves GPTQ by nearly 10%. Both facts shape our baselines below.
- The HDI paper shows that low HDI can reflect calibrated refusal rather than confabulation. For example, GPT-5.4-mini's 0.336 came with a 3.2% semantic-overlap probe result, which the authors attribute to refusal. We therefore do not rely on HDI alone.
Research Program
The first component is a measurement. For a note $r$ and question $q$ with correct answer $y$, we create an edited note $\tilde r$ that changes one clinically meaningful value, with recomputed answer $\tilde y$.
- On MedCalc-Bench, the edits are to calculator inputs and the formula supplies $\tilde y$.
- On TCGA-LUAD bundles, the edits are to stage, smoking history or expression values. Expected claim changes are adjudicated against the edited bundle.
Counterfactual tracking rate is the fraction of pairs where the model answers $y$ on $r$ and $\tilde y$ on $\tilde r$. Rationale tracking checks whether the rationale cites the edited value rather than the original or a generic default. A model that has stopped reading the record will answer correctly on unedited notes whose values sit near population priors, and will fail the edited ones.
The second component is the protection method. We define the record-dependence margin as
$$ D_\theta(r,\tilde r) = \big[\log p_\theta(y \mid r,q) - \log p_\theta(y \mid \tilde r,q)\big] + \big[\log p_\theta(\tilde y \mid \tilde r,q) - \log p_\theta(\tilde y \mid r,q)\big], $$
where $p_\theta$ is the model's probability of an answer given note and question. $D$ is large only when the answer moves with the record. We replace TACQ's task-loss target with the expected drop in $D$ under simulated quantization, keeping its magnitude-sharpened gradient term.
We then preserve the top-p% of weights in 16-bit, ranked by this score, and quantize the rest with GPTQ. Pairs for computing saliency come from a held-out split of MedCalc-Bench calculators and synthetic record edits. They are kept disjoint by calculator type from the evaluation pairs, so the method cannot memorize test formulas.
The third component is a localization analysis. It asks whether record-sensitive weights cluster in the final layers, as the "last layer" finding predicts, and how much they overlap with TACQ's task-loss weights. Low overlap would itself be a finding: accuracy and grounding would then rely on partly different parameters.
Baselines
The naive reference points are the 16-bit model and plain round-to-nearest quantization at each bit width. Together they bound the range of possible damage.
Off-the-shelf quantizers. These are GPTQ, AWQ and BitsAndBytes at 8, 4 and 3 bits. AWQ is reported to be more robust than GPTQ for weight-only reasoning in Quantization Hurts Reasoning?, and BitsAndBytes best preserved knowledge fidelity in Through a Compressed Lens. These are what a hospital would actually deploy, so any method we propose must beat the best of them.
TACQ with task-loss saliency. This is the strongest existing competitor. It uses the same protected-weight fraction and the same GPTQ backbone, and its saliency is computed on the same clinical notes using answer cross-entropy instead of $D$. It is therefore the direct ablation of our key ingredient. The only reported number we have for it is 96% MMLU recovery at 3.1 bits on Llama-3-8B-Instruct, from the TACQ paper. We found no reported result on Lacuna for TACQ, or any quantized model, on MedCalc-Bench, TCGA-LUAD HDI or counterfactual record tracking.
Clinically calibrated GPTQ and AWQ. These are calibrated on the same clinical notes our method sees, which isolates whether the saliency objective adds anything beyond domain data. Last-layer protection keeps the final layers in 16-bit at a matched average bit budget.
For full-precision references we use the MedCalc-Bench reports of 50.9% (GPT-4, one-shot chain of thought) and 39.5% (Llama 3-70B). No numbers are reported for our exact open models, so we measure them ourselves.
All methods share the same models (Llama-3-8B-Instruct, Qwen2.5-7B and 14B, and DeepSeek-R1-Distill-Qwen-7B), calibration and saliency data, and decoding settings. They are compared at matched effective bits, including the memory cost of protected weights. Each gets the same budget for choosing p and the calibration size.
Our method beats the strongest baseline if it meets three conditions. At 3 bits it must exceed the best competitor on counterfactual tracking by a margin larger than the seed-to-seed spread. It must not lose more than one point of unedited accuracy. It must hold the same direction on TCGA-LUAD, where no saliency pairs were drawn.
Why not just…?
Why not just use 8-bit, which is already "lossless"? If 8-bit preserves counterfactual tracking, that is a useful negative result and we will report it as the deployment recommendation. However, the multi-agent clinical pipelines that motivate local deployment, such as the five-role system in A Multi-Agent Framework for Interpreting Multivariate Physiological Time Series, multiply memory per patient. That pressure pushes toward 4 bits and below. The "lossless" verdict has also only been checked on final answers, which is exactly the metric we doubt. Our sweep includes 8-bit, so the claim is tested rather than assumed.
Why not just calibrate standard quantizers on clinical notes? This is the most serious threat. Domain calibration gave nearly 10% for GPTQ on reasoning tasks, and it may restore record-reading on its own. Our clinically calibrated baseline uses identical data, so any gain from the $D$ objective must exceed what the data alone provides. If it does not, we will conclude that the measurement, not the method, is the contribution.
Why not just protect the last layers? Through a Compressed Lens points there, and this is a one-line heuristic. It is weaker in principle because extraction failures happen when the note is read, which may involve middle-layer attention rather than the output layers. Our localization analysis and the matched-budget baseline settle this directly.
Why not just route calculations to a code tool? Tools fix arithmetic. Vivaldi's code-running agent eliminated math hallucination, but a calculator still needs correctly extracted inputs, and extraction is where roughly half of MedCalc-Bench failures occur. We also evaluate every method with an external calculator, which isolates extraction-level record-reading from arithmetic.
Decisive Checks
The first check tests the core hypothesis. Across bit widths, does counterfactual tracking fall faster than unedited accuracy under off-the-shelf quantization? If the two fall together, the problem we target does not exist for weight quantization, and the method is moot.
The second check tests the key ingredient. At matched bits, does $D$-based saliency beat task-loss TACQ and clinically calibrated GPTQ on held-out calculators and on TCGA-LUAD? We also report HDI together with refusal rates and the semantic-overlap probe, so that refusal cannot masquerade as grounding.
The third check is a small blinded clinician review of 3-bit rationales on edited notes. It asks whether rationales from the protected model cite the edited value. This guards against automatic judges, which Can Large Language Models Still Explain Themselves? found fail to track human-perceived degradation.
References
- Does Accuracy Equal Evidence? Reasoning Faithfulness under KV Cache Compression
- Can Large Language Models Still Explain Themselves? Investigating the Impact of Quantization on Self-Explanations
- Through a Compressed Lens: Investigating the Impact of Quantization on LLM Explainability and Interpretability
- The Hallucination Dependence Index: A Cross-Condition Diagnostic for Clinical-LLM Faithfulness
- MedCalc-Bench: Evaluating Large Language Models for Medical Calculations
- Task-Circuit Quantization: Leveraging Knowledge Localization and Interpretability for Compression
- Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models
- A Multi-Agent Framework for Interpreting Multivariate Physiological Time Series