Counterfactual Twin Calibration for Bias-Neutral Low-Bit LLMs

Shrinking a large language model's weights from 16-bit numbers to 2 or 3 bits makes it several times cheaper to serve. The methods that work best at these extreme settings decide which weights deserve extra precision by looking at a small "calibration" set of example text. We propose making that calibration text out of counterfactual twins: pairs of fluent, model-generated passages that are identical except for the social group they describe. We then protect the weights that control how differently the model treats each twin. The goal is modest and checkable. A compressed model should be no more biased than the full-precision model it came from, at the same bit budget and the same inference cost.

The reason to expect a problem is a mismatch between what compression protects and what fairness depends on. Post-training quantization rounds each weight to a coarse grid. It is usually tuned to keep the model's average loss on generic text low, and a fully trained model sits near a minimum of that loss, so small rounding errors hurt it only at second order. The difference in log-likelihood the model assigns to a sentence about one group versus the same sentence about another group is not at any minimum. Rounding noise therefore moves it at first order. Compression can shift a model's treatment of groups much faster than it shifts perplexity, and a calibration set of generic web text gives the quantizer no signal about that shift. Our method supplies that signal cheaply and on-distribution.

Core Claim

At sub-3-bit precision, saliency-guided quantization makes bias drift worse, as a side effect of how it chooses which weights to keep in full precision. Bias drift here means the change in a model's bias scores relative to its own full-precision version. The drift can be sharply reduced, without extra inference cost and with at most a small loss in task accuracy, by adding one term to the saliency score. That term measures how strongly each weight controls the log-likelihood gap between counterfactual twins. The twins are generated by the full-precision model itself and rewritten so the whole context stays coherent, not just the swapped identity word.

For a twin pair $(x, x')$, let $\Delta(w) = \log p_w(x) - \log p_w(x')$ be the likelihood gap under weights $w$. For a quantization perturbation $\delta$, a first-order Taylor expansion gives $\Delta(w+\delta) - \Delta(w) \approx \nabla_w \Delta \cdot \delta$. We therefore score each weight $w_i$ by

$$ s_i = \mathbb{E}_{x}\big[(\partial_{w_i} \mathcal{L})^2\big] + \lambda\, \mathbb{E}_{(x,x')}\big[(\partial_{w_i} \Delta)^2\big], $$

where $\mathcal{L}$ is the usual language-modeling loss on calibration text and $\lambda$ trades task fidelity against bias preservation. The first term is the Fisher-style saliency used by existing methods. The second term is new. It is the expected squared first-order change in the twin gap, so it identifies the weights whose rounding would most move the model's relative treatment of groups. The top-scoring weights are kept at 16 bits, and the rest are quantized with the standard rounding procedure, using a Hessian averaged with equal weight across groups.

Where This Sits in Prior Work

Several papers motivate the problem without solving it. Does Compression Exacerbate Large Language Models' Social Bias? measures likelihood bias on HolisticBias templates after off-the-shelf pruning and quantization. It finds that LLaMA models become markedly more biased, with 57–64% on the out-of-distribution "Nonce" axis, but it tests no remedy. Unmasking the Hidden Fairness, Bias, and Safety Costs of Compression with Mixture-of-Expert Models finds that 4-bit and 8-bit quantization is the most "neutral" compression for mixture-of-experts models. It also finds that swapping calibration data for human-written safety text can increase bias on BBQ, the question-answering benchmark for social bias. And it warns of a "mirage of fairness", where collapsed models look unbiased because they fail uniformly. That paper constrains us in two ways. It suggests the effect may be small at 4 bits, so we focus on 2–3 bits. It also shows that naive data mixing is not enough, so our calibration data must be constructed rather than filtered.

The theory we build on comes from vision. Understanding the Unfairness in Network Quantization expands group-specific losses to second order and shows that group gradient norms and Hessian traces govern how much each group is harmed. It also finds that balancing the training data largely closes the gap. It studies classifiers with labeled groups, not language models or calibration sets. FairQuant learns per-layer bit widths against a demographic-parity objective for medical image classifiers, but it needs patient demographic labels during quantization. Our first-order twin-gap term moves both ideas to language generation, where group labels exist only implicitly in text, and replaces label access with generated pairs.

The saliency machinery comes from compression methods. Task-Circuit Quantization (TACQ) keeps the most task-salient weights at 16 bits and quantizes the rest to 2–3 bits. GuidedQuant weights the rounding error by gradients of the model's final loss, approximated with a block-averaged Fisher matrix. Both compute saliency on aggregate data such as MMLU, GSM8k, or C4, where majority-group text dominates. That dominance is exactly the mechanism we suspect drives bias drift. We keep their machinery and change only what the saliency is measured on and against.

The data-generation side builds on Self-calibration for Language Model Quantization and Pruning. That work has the model generate its own calibration text starting from an empty prompt and finds that 16 self-generated examples can beat 128 C4 examples, but it never measures bias. It also builds on the synthetic-debiasing observation that flipping a protected attribute alone produces "off-manifold" samples, meaning statistically incoherent examples that real data would never contain (Li et al.; Celis et al.). It follows Downstream Fairness Transfer via Fair Data Synthesis in placing the fairness intervention in the data rather than in the downstream objective. GenQ shows that generated calibration data works for low-bit vision models, but it generates by class label with no attention to social groups.

Finally, Multi-Feature Quantized Self-Attention (MQAR) uses a codebook of discrete prototype vectors inside a full-precision model to strip demographic information from attention. It compresses nothing. Our work is its complement: it changes no architecture and aims at physical compression.

Research Program

We will build twin sets from the full-precision model itself. The model is seeded with short, neutral scene prompts that mention a person from one group. It continues the text at moderate temperature, following the temperature schedule of Williams et al. It is then asked to rewrite the passage for a counterpart group, adjusting names, pronouns, and dependent context so the result stays coherent.

We accept a twin only if two conditions hold. First, its per-token likelihood under the full-precision model is within a tolerance of the original, which keeps it on-manifold. Second, a string-difference check confirms the change is confined to identity-bearing spans. Twins cover a training subset of demographic axes: gender, race and ethnicity, and religion. Age, disability, nationality, and the Nonce axis are held out, so we can test whether the protection generalizes rather than memorizes a benchmark. Twin seeds are drawn from templates disjoint from BBQ, CrowS-Pairs, StereoSet, and HolisticBias, and we will check n-gram overlap to rule out contamination.

Calibration mixes about 128 generic self-generated passages with about 128 twin pairs. We compute the balanced Hessian and the two saliency terms, keep the top 0.3–1% of weights at 16 bits so that effective bit width stays matched to TACQ's roughly 3.1-bit setting, and quantize the rest with the same rounding routine TACQ uses. We will also run the procedure on top of QuIP-style incoherence processing (QuIP), which spreads large-magnitude weights across dimensions with random rotations before rounding. This tests whether the twin term still matters when the rounding itself is far more accurate.

The computational-cost direction enters as a hard constraint. Every compared model has the same number of stored bits and the same kernel, so any bias reduction comes at zero serving cost. We report the calibration overhead in GPU-hours: one extra backward pass per twin pair.

The analysis component tests the mechanism directly. For each quantized model we compute the predicted first-order gap shift $\nabla_w \Delta \cdot \delta$ and correlate it with the measured shift on held-out twins. If the correlation is weak, the first-order story is wrong and the saliency term is unjustified, whatever the benchmark numbers say.

Baselines

The naive reference points are the full-precision model and uniform rounding with generic C4 calibration. The full-precision model is the target: bias drift is measured as the absolute change from it. Uniform rounding uses GPTQ and QuIP with the standard 128 C4 samples. For scale, QuIP reports 6.33 WikiText-2 perplexity on Llama 2 70B at 2 bits, while the rounding method OPTQ collapses to 123.91 (QuIP).

Self-calibration (Williams et al.) is the closest data-free competitor. It is run with its published temperature schedule and the same number of passages.

The strongest existing method is TACQ (Task-Circuit Quantization). It reports recovering 96% of unquantized MMLU performance on Llama-3-8B-Instruct at 3.1 bits, a 5.25% absolute improvement over SpQR, and 87.93% of MMLU on Qwen2.5-32B-Instruct at 2 bits. GuidedQuant (Kim et al.) is a second strong competitor, reporting 8.83 WikiText-2 perplexity on Llama-2-7B at 2 bits, down from 39.58. We found no reported result on Lacuna for either method on BBQ, CrowS-Pairs, StereoSet, or HolisticBias, so we will produce these numbers ourselves.

We use the released TACQ and GuidedQuant procedures, the same models, the same fraction of 16-bit weights, and the same number of calibration tokens. Each method gets an equal hyperparameter-search budget of eight configurations, selected on a validation split disjoint from all test benchmarks.

We add a safety-mix baseline replicating the 11.8% and 100% tulu3 safety-data settings of Szentmiklossy et al.. We also add a label-based group reweighting in the spirit of FairQuant, in which the loss saliency is averaged with equal weight over group-tagged text but no twin-gap term is used.

Three ablations isolate the key ingredients. Twin data with $\lambda = 0$ tests whether balanced data alone does the work. Token-swapped twins, with the identity word changed and nothing else, test whether on-manifold rewriting matters. The twin-gap term computed on held-in axes only, evaluated on held-out axes, tests generalization.

Beating TACQ means two things together. First, bias drift must fall by at least half, averaged over BBQ, CrowS-Pairs, and HolisticBias, with the difference exceeding the spread across 5 calibration seeds. Second, MMLU, GSM8k, and zero-shot averages must stay within one point of TACQ.

Why not just…?

Why not just calibrate with token-swapped counterfactuals and skip the generation step? This is the most serious threat. Swapping a name or pronoun may change a model's internal activations enough to steer the quantizer, and it costs nothing. The synthetic-debiasing literature argues that isolated attribute flips produce incoherent samples (Li et al.), but that argument comes from tabular data and may not transfer to text. The token-swap ablation settles the question directly. If it matches the full method on held-out axes, we will report that the generation step is unnecessary and recommend the cheaper recipe.

Why not just stop at 4 bits, where quantization already looks neutral? Szentmiklossy et al. do find AWQ and GPTQ closest to the dense baseline, though on mixture-of-experts models. Does Compression Exacerbate… nonetheless finds significant drift for dense LLaMA models, and the serving cost is what motivates 2–3 bits. Our bit-width sweep will show where drift begins. If baseline drift is negligible even at 2 bits, the problem does not exist, and we will say so.

Why not just calibrate on human-written safety or fairness text? The only direct test found on Lacuna reports that 100% safety calibration can increase BBQ bias (Szentmiklossy et al.). Safety text teaches refusal, not group symmetry. We include this setting as a baseline so the comparison is empirical rather than asserted.

Why not just debias the compressed model afterwards with a plug-in such as MQAR? A plug-in adds a module to the model and would need retraining for each compressed variant. Its bias reductions are also measured against a full-precision backbone (MQAR). The two approaches can be combined. We will apply MQAR-style debiasing on top of both TACQ and our method, to see whether preventing drift during compression still helps after post-hoc debiasing.

Proposed Experiments

Models are Llama-2-7B (matching GuidedQuant), Llama-3-8B-Instruct (matching TACQ), and Qwen2.5-14B. We quantize at 4, 3, roughly 3.1 with mixed precision, and 2 bits, with 5 calibration seeds each, because calibration volatility can swing downstream accuracy by nearly 10%.

Fairness is measured on BBQ (Parrish et al.), with ambiguous-context bias and disambiguated-context accuracy always reported together to expose the mirage of fairness. We also use CrowS-Pairs (Nangia et al.), StereoSet, and HolisticBias likelihood bias with Mann-Whitney tests. Capability is measured with MMLU, GSM8k, the 10-task zero-shot suite, and WikiText-2 perplexity.

The primary metric is absolute drift from the full-precision model, summed across axes, reported separately for held-in and held-out axes.

Three results would decide the proposal. First, baseline drift must grow as bit width falls and must be larger under saliency-guided methods than under uniform rounding. That pattern is our proposed mechanism. Second, predicted first-order gap shifts must correlate with measured shifts on held-out twins, with Spearman correlation above 0.5. Third, the full method must beat both TACQ and the token-swap ablation on held-out-axis drift at matched accuracy and matched stored bits. The proposal fails outright if the token-swap ablation matches it, or if the drift reduction appears only on the axes used to build the twins.