Compressing a large language model to fit a memory budget usually means storing its weights with fewer bits. Newer "fractional" quantizers can hit any average budget, such as 2.4 or 2.875 bits per weight, by giving some layers more bits than others. They decide which layers to protect using perplexity or weight reconstruction error, never fairness. Separately, fairness researchers have established that a model's internal bias (stereotyped associations visible in its token probabilities or hidden states) barely predicts its external bias (unequal behavior on real tasks like question answering or coreference). We propose to join these two facts with one tool. Fractional quantizers let us lower the precision of a single layer, a controlled and graded intervention that full-precision debiasing studies cannot make. We use that intervention to build a per-layer map of where internal and external bias live. We then test whether a bit allocator guided by external fairness sensitivity keeps deployed models fairer, at the same memory budget, than one guided by perplexity or by internal bias probes.
The practical stake is concrete. The deployments these quantizers target are exactly where fairness can quietly degrade: a 70B model squeezed into 24GB of GPU memory at 2.4 bits by LiftQuant, or 2.875-bit models from Q-Palette. Integer quantization is already known to raise decision-level disparity. The scientific stake is that the intrinsic–extrinsic gap has so far been measured only by correlating metrics across different models. Per-layer precision reduction turns it into a within-model question about where each kind of bias is stored.
Core Claim
Our central hypothesis has two parts. First, layers whose precision reduction most changes internal bias metrics are largely not the layers whose precision reduction most changes downstream disparity. If so, the representation–behavior disconnect holds at the level of individual layers, not only across models. Second, a fractional allocator that spends bits according to a capability-controlled measure of downstream disparity will preserve downstream fairness better than two alternatives at matched average bit-width and matched perplexity. The alternatives are perplexity-driven allocation and allocation driven by internal probes.
The "capability-controlled" qualifier is essential. How Quantization Shapes Bias in Large Language Models found an "uncertainty trap": quantized models look less biased on probability metrics only because their output entropy rises. Unmasking the Hidden Fairness, Bias, and Safety Costs of Compression with Mixture-of-Expert Models found a "mirage of fairness," where BBQ bias scores improve because the model fails uniformly. Any sensitivity signal that ignores these effects would reward the allocator for breaking the model.
Relation to Prior Work
Two lines of work motivate the proposal by documenting the gap. On the Intrinsic and Extrinsic Fairness Evaluation Metrics for Contextualized Language Representations correlated intrinsic and extrinsic metrics across 19 models. It found negligible correlation, 0.21 for gender toxicity with CEAT, rising to 0.91 only when the intrinsic test was redesigned to match the downstream notion of bias. Measuring Fairness with Biased Rulers showed a similar lack of alignment between SEAT and CrowS-Pairs on one side and Bias in Bios and WinoBias on the other. Both studies compare across models, whereas we compare across layers inside one model under a graded intervention.
How Gender Debiasing Affects Internal Model Representations introduced minimum-description-length probing, which measures how compressibly a hidden state encodes gender. It showed that downstream bias can fall while this internal encoding stays intact. We reuse its probe as one of our internal measures.
A second group constrains the design. Linear Guardedness and its Implications shows that erased concepts leak back through multiclass decision boundaries, so internal "erasure" cannot certify downstream behavior. That result argues against allocating bits by internal geometry alone. The uncertainty-trap and mirage-of-fairness findings force us to condition every disparity measure on capability.
Three papers are the closest methods. FairQuant allocates 2/4/8-bit precision in medical image classifiers using Taylor estimates of per-group loss change, inside quantization-aware training. It never touches language models, fractional post-training quantizers, or the intrinsic–extrinsic distinction, and its group-loss signal measures group damage rather than change in disparity. Alignment-Aware Quantization for LLM Safety changes the calibration loss to preserve refusal behavior, not social bias, and does not change bit allocation. Multi-Feature Quantized Self-Attention uses "quantization" in the codebook sense, as a debiasing bottleneck, and is unrelated to memory compression.
The quantizers supply the machinery. Q-Palette provides fractional per-layer quantizers and a knapsack allocator. Variable Layerwise Quantization finds a U-shaped importance curve that compresses middle layers hardest. We aim to find out whether that U shape is fairness-safe.
Research Program
Stage one: the layer map. We take LLaMA-3.1-8B and Qwen-2.5-7B, the families studied in How Quantization Shapes Bias, so our integer reference points are comparable. We start from a 4-bit Q-Palette base. For each transformer block in turn, we drop that block alone to 3.0, 2.5, and 2.0 bits and leave the rest at 4.0. For each of these single-block interventions we record five kinds of change:
- Perplexity on WikiText-2 and C4.
- Output entropy.
- Internal bias: CrowS-Pairs, StereoSet, the HolisticBias likelihood metric from Does Compression Exacerbate Large Language Models' Social Bias?, and minimum-description-length gender probes on hidden states.
- Downstream disparity on development splits.
- BBQ disambiguated accuracy, as a capability measure.
The capability-controlled downstream disparity uses three parts of the development data. On BBQ, it is the accuracy difference between disambiguated items whose correct answer matches a stereotype and items whose answer contradicts one. The original BBQ paper reports this gap as 3.4 points for full-precision models. We add the ambiguous-context rate of stereotyped answers among non-"unknown" responses. On WinoBias, it is the pro- versus anti-stereotypical accuracy gap.
The primary analysis is the rank correlation, across blocks, between internal sensitivity and downstream sensitivity. We test additivity on a random subset of block pairs. If pairwise interactions are large, the allocator must model them rather than sum per-block costs.
Stage two: the allocator. Let $b_l$ be the bit-width of block $l$, drawn from Q-Palette's palette. Let $n_l$ be the block's parameter count, $N$ the total parameter count, and $B$ the target average bits. Let $D_l(b_l)$ be Q-Palette's distortion cost and $S_l(b_l)$ the measured capability-controlled disparity increase from stage one. We solve
$$\min_{\{b_l\}} \sum_l D_l(b_l) + \lambda \sum_l S_l(b_l) \quad \text{subject to} \quad \sum_l n_l b_l \le B N,$$
and choose $\lambda$ as the largest value that keeps WikiText-2 perplexity within a fixed tolerance of the perplexity-only solution. This keeps utility matched by construction. For LLaMA-3-70B at the 2.4-bit budget, we measure sensitivity on groups of four blocks, to control the depth-linear cost that the quantization literature flags as an open problem.
Baselines
The naive reference points are FP16 and uniform integer quantization with GPTQ and AWQ at 3 and 4 bits. These reproduce the setting of How Quantization Shapes Bias, which reports Demographic Parity Difference on DT-Fairness rising by over 7 points under SmoothQuant W4A8 for some LLaMA variants. They tell us whether fractional budgets fall on the harmful or benign side of integer cliffs.
The strongest competitor on this question is unmodified Q-Palette, a perplexity/distortion knapsack allocator. It reports that its 2.875-bit models beat prior 3.25-bit models in accuracy. No reported fairness result was found on Lacuna for Q-Palette, or for any fractional quantizer, so its fairness numbers must be produced here. It is the right opponent because the do-nothing hypothesis is that utility-optimal allocation already protects fairness. We run it with the same quantizers, calibration data, budgets (2.4, 2.75, 2.875, 3.25 bits), and solver. We also run Variable Layerwise Quantization's importance ordering, which uses layer input modification (how much a layer changes its input), as a second utility-driven allocator.
The strongest fairness-aware competitor is a port of FairQuant's signal. Each block's cost is the first-order Taylor estimate of per-demographic-group loss increase on the same development data, fed into the same knapsack. FairQuant reports recovering ResNet18 accuracy on Fitzpatrick17k from 23.4% to 45.33% at about 4 bits, but that is a vision number and does not transfer. It receives the same development data and $\lambda$ tuning budget as our allocator.
The key ablation replaces $S_l$ with internal sensitivity, built from CrowS-Pairs, StereoSet, and the probes, at identical tuning. A second ablation removes the capability control and uses raw BBQ bias score. That tests whether the allocator then learns to buy "fairness" by damaging accuracy.
Our method beats the field if it cuts the fractional model's downstream disparity increase over FP16 on held-out benchmarks by a clear margin relative to Q-Palette. The pre-registered target is at least half of that increase removed, averaged over budgets. It must do so at matched perplexity and without lower BBQ disambiguated accuracy, and it must outperform both the Taylor-group-loss and intrinsic-guided ablations.
Why not just…?
Why not just trust the perplexity-optimal allocator? This is the real threat. If fairness-relevant computation sits in the same layers perplexity already protects, stage one will show high rank correlation between perplexity sensitivity and disparity sensitivity. Our allocator will then converge to Q-Palette's solution. We treat that as a publishable negative result that clears current fractional quantizers, and we will report it as such.
Why not just calibrate with fairness data? Unmasking the Hidden Fairness, Bias, and Safety Costs found that safety-heavy calibration did not restore fairness and, at 100% safety data, sometimes increased BBQ bias. That study covered mixture-of-experts models with integer quantizers only. We therefore include a dense-model control: Q-Palette calibrated on a mix of BBQ and WinoBias development text, with perplexity allocation. It separates the effect of where bits go from what the quantizer is fit to.
Why not just use inference-time prompting? BiasFreeBench reports Chain-of-Thought reaching a 96.11% Bias-Free Score on BBQ, well above weight-level methods. Prompting is complementary, not competing. It adds tokens and passes on the memory-limited hardware these quantizers target. Bias after Prompting reports that bias persists under prompt adaptation. We evaluate every allocator both with and without a Chain-of-Thought prompt. If prompting alone erases the gap between allocators on generation benchmarks, we will say so. We expect it to matter less on DT-Fairness and DiscrimEval decision tasks.
Why not just keep sensitive layers at 4 bits by intuition, for example the first and last? The U-shaped importance curve already does this. Our stage-one map directly tests whether disparity sensitivity is concentrated in the middle blocks that curve compresses.
Decisive Checks
To guard against optimizing one metric at the expense of others, sensitivity is measured on BBQ development categories and WinoBias development data. Final evaluation uses held-out BBQ categories, DT-Fairness and DiscrimEval Demographic Parity Difference, and BOLD generation toxicity and sentiment. We also report the internal metrics and zero-shot utility (MMLU, ARC, HellaSwag) at every budget, and repeat everything on three quantization seeds.
Four outcomes decide the claims:
- A low within-model rank correlation between internal and downstream layer sensitivity would extend the intrinsic–extrinsic gap to the layer level.
- A win for extrinsic-guided allocation over intrinsic-guided allocation on held-out tasks would show that internal probes are poor guides for where to spend bits.
- A win over the Taylor group-loss port would show that disparity change, not group damage, is the right signal.
- Non-monotone behavior across the fine budget sweep from 4.0 to 2.4 bits would locate phase transitions that integer studies step over.
If the capability-uncontrolled ablation "wins" only by lowering disambiguated accuracy, that confirms the mirage-of-fairness effect and justifies the control.
References
- LiftQuant: Continuous Bit-Width LLM via Dimensional Lifting and Projection
- Q-Palette: Fractional-Bit Quantizers Toward Optimal Bit Allocation for Efficient LLM Deployment
- How Quantization Shapes Bias in Large Language Models
- Unmasking the Hidden Fairness, Bias, and Safety Costs of Compression with Mixture-of-Expert Models
- On the Intrinsic and Extrinsic Fairness Evaluation Metrics for Contextualized Language Representations
- Measuring Fairness with Biased Rulers: A Comparative Study on Bias Metrics for Pre-trained Language Models
- How Gender Debiasing Affects Internal Model Representations, and Why It Matters
- Linear Guardedness and its Implications
- FairQuant: Fairness-Aware Mixed-Precision Quantization for Medical Image Classification
- Alignment-Aware Quantization for LLM Safety
- Multi-Feature Quantized Self-Attention for Fair Large Language Models
- Variable Layerwise Quantization: A Simple and Effective Approach to Quantize LLMs
- Does Compression Exacerbate Large Language Models' Social Bias?
- BBQ: A hand-built bias benchmark for question answering
- BiasFreeBench: a Benchmark for Mitigating Bias in Large Language Model Responses
- Bias after Prompting: Persistent Discrimination in Large Language Models