Teams that release a multilingual language model often remove some of its stereotypes first, usually by training or editing in English and relying on the change to carry over to French, German or Hindi. The model is then compressed. Its weights are rounded to 2–4 bits, a step called post-training quantization, so that it fits on cheaper hardware. Nobody checks whether the English-trained fairness edit still reaches the other languages after this rounding. The compression methods judge which weights to protect by English perplexity, MMLU, or a task loss. The weights that carry a debiasing effect across languages are probably few and low in magnitude. That is exactly what aggressive rounding destroys.
This note proposes two things. The first is a measurement: how much of a cross-lingual debiasing effect survives 2–4 bit quantization and low-rank compression, broken down by language. The second is a method that targets the failure if the measurement finds one. It is a quantizer that decides which weights to keep at full precision using a cross-lingual stereotype objective as well as a task objective. The central prediction is that compression erodes debiasing unevenly. English retains most of the effect, while typologically distant languages such as Hindi lose disproportionately more. A fairness-aware choice of protected weights, under a fixed bit budget, should close most of that gap.
Core Claim
Cross-lingual debiasing works because languages share internal directions in the model. Prior work established this transfer at full precision. An autoencoder-aligned latent space gives "up to 65%" bias reduction on multilingual CrowS-Pairs in Debiasing Multilingual LLMs in Cross-lingual Latent Space. English gradient-ascent unlearning on Aya-Expanse-8B cuts stereotypical answers on SeeGULL "from 16% to 3% in English, 13% to 6% in French, and 13% to 7% in Hindi" in UNLEARNING GEO-CULTURAL STEREOTYPES IN MULTILINGUAL LLMS. An English adapter lowers German BEC-Pro bias from 1.10 to 0.67 zero-shot in Sustainable Modular Debiasing of Language Models. These papers motivate the question but never compress the model.
The compression literature supplies the method and a warning. Task-Circuit Quantization, called TACQ, keeps the top-$p\%$ of weights in 16-bit precision. It picks them by a gradient-based estimate of how much rounding each weight would change a chosen loss. It reports that capabilities absent from that loss degrade sharply at 2 bits. ASVD shows that activation outliers dominate which directions survive low-rank truncation, and it calibrates on 32 English Wikitext samples. Our claim is that the cross-lingual debiasing effect is one such unprotected capability. Making it part of the saliency loss is cheap and sufficient.
Research Program
The pipeline follows realistic deployment: debias first in full precision using English data only, then compress. We study three debiasing interventions that span the main mechanisms in the file. The first is English gradient-ascent unlearning with retain and KL terms, following the geocultural unlearning paper. The second is weight-free steering by an unsupervised concept vector, following Unsupervised Concept Vector Extraction for Bias Control in LLMs. The third is projection debiasing in a cross-lingual latent space, following Peng et al. We cross these with four compressors at matched memory budgets: GPTQ, SPQR, TACQ, and ASVD at parameter ratios 0.9 and 0.75.
The models are Aya-Expanse-8B and Llama-3.1-8B, the pair whose divergent Hindi behavior was documented under unlearning, plus Llama-3-8B-Instruct, where TACQ's numbers were reported.
The primary quantity is a per-language retention ratio:
$$R(l)=\frac{B_{0}(l)-B_{dc}(l)}{B_{0}(l)-B_{d}(l)},$$
The symbols are defined as follows:
- $B_0(l)$ is the bias score of the original model in language $l$.
- $B_d(l)$ is the bias score after full-precision debiasing.
- $B_{dc}(l)$ is the bias score of the debiased model after compression.
Here $R=1$ means the edit fully survives and $R\le 0$ means it is erased. On CrowS-Pairs, bias is measured as distance from the 50% preference point rather than raw stereotype preference. This matters because Investigating Bias in Multilingual Language Models found that English debiasing can overshoot into counter-stereotype preference. A metric that rewards lower preference would hide that overshoot.
The method, fairness-aware saliency, keeps TACQ's machinery and changes what it protects. Weight importance is computed on the loss
$$\mathcal{L}=\mathcal{L}_{\text{task}}+\alpha\sum_{l\in S}\mathcal{L}_{\text{bias}}(l),$$
The symbols are defined as follows:
- $\mathcal{L}_{\text{task}}$ is TACQ's usual calibration loss.
- $\mathcal{L}_{\text{bias}}(l)$ is the squared gap between the debiased full-precision model's and the simulated-quantized model's log-likelihood margin on stereotype versus anti-stereotype pairs in language $l$.
- $S$ is the set of calibration languages.
- $\alpha$ is a weight tuned on held-out pairs.
We test two settings for $S$. In the first, $S$ is English only. In the second, $S$ adds machine-translated pairs in a few languages. The question is whether protecting English bias behavior alone is enough to preserve transfer. If it is, the shared geometry survives once its anchors are kept. If it is not, compression breaks the alignment itself.
Two further analyses explain whatever we find. We measure the cosine similarity of the steering direction extracted per language before and after compression. We also measure what fraction of the TACQ-protected weights overlap with weights that carry high bias-loss gradient. Both tell us whether the damage lands on the shared subspace or on language-specific pathways.
The domain and noise directions enter as constraints on deployment, not as core mechanisms. Compression is known to hurt rare knowledge first. We therefore check that the fairness term does not buy fairness at the cost of specialist accuracy, using translated emrQA Medication from When Multilingual Models Compete with Monolingual Domain-Specific Models in Clinical Question Answering in Spanish, Polish and Greek. We also run a small distractor stress test on Natural Questions with 20 retrieved passages. This checks that low-bit models with protected bias weights are not more easily misled by irrelevant context, a failure documented for uncompressed readers in Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents.
Baselines
The naive reference point is the debiased full-precision model, which gives the ceiling $R=1$. Beside it sit two other reference points. One is the original undebiased model compressed with GPTQ at the same bit width, which tells us what $B$ looks like with no fairness edit at all. The other is the debiased model compressed with plain GPTQ on English Wikitext calibration. This last configuration is what practitioners actually ship, and it defines the gap we must close.
The strongest existing compressor is TACQ with its standard task-loss saliency. The file reports that on Llama-3-8B-Instruct it recovers "96% of the unquantized MMLU performance with only 3.1 bits, a 5.25% absolute improvement over SPQR." At 2 bits it reports "14.74% over the strongest baseline, SliM-LLM." It is the right competitor because it already protects a small set of task-critical weights. Our proposal differs only in what loss defines "critical." We run it with identical bit budgets, the same protected fraction $p$, the same number of calibration tokens, and the same GPTQ backend.
No reported result found on Lacuna for any bias benchmark under quantization or low-rank compression. The comparison point for bias is therefore the full-precision debiased numbers above, chiefly the Aya-Expanse SeeGULL figures. Beating TACQ means a higher mean $R(l)$ across non-English languages, especially Hindi, at equal bits and equal protected fraction. It also requires MMLU and emrQA F1 within one point of standard TACQ.
The key ablation separates the two ingredients. The first variant uses multilingual calibration text with no bias term. The second uses the bias term on English pairs only. The third uses the bias term on multilingual pairs. If the first variant alone recovers transfer, the contribution is calibration language, not fairness-aware saliency, and we will report it that way. A further control spends the bias-term's protected weights on randomly chosen weights instead. This rules out the possibility that any extra full-precision budget helps.
Why not just…?
Why not just debias after quantizing? Re-running unlearning or adapter training on the compressed model is the obvious repair. It requires full fine-tuning infrastructure for low-bit weights, which most downstream quantizers lack, and it must be redone for every compressed variant. The geocultural unlearning paper also shows that gradient ascent can collapse Llama-3.1-8B into "random, meaningless responses" in Hindi. We expect that risk to be worse on noisier low-bit weights. We include post-quantization unlearning, run with QLoRA-style adapters, as a comparison arm at matched compute. If it matches our retention without collapse, the method's value reduces to cost, and we will say so.
Why not just re-extract the steering vector on the compressed model? This is the most serious threat for the steering intervention. Vector extraction in the unsupervised concept-vector paper needs no labels and costs only forward passes. We run it directly on each compressed model. Re-extraction cannot repair edits baked into the weights, such as unlearning or projection, which is where we expect the method to matter. If re-extraction fully restores cross-lingual transfer for steering, that is itself a finding: the shared bias direction survives compression and only its magnitude drifts. We will then narrow our claim to weight-level edits.
Why not just calibrate with multilingual text? Wikitext-only calibration is plausibly the whole problem. Multilingual calibration is the first ablation variant in the Baselines section. The prediction that separates the two explanations is this. Generic multilingual text preserves perplexity and MMLU in all languages, yet still lets $R(\text{hi})$ fall. The reason is that the debiasing edit is a small, deliberate deviation from pretraining statistics, and generic text does not reward keeping it.
Proposed Experiments
The decisive experiment is the full grid of three interventions, four compressors, three bit widths (4, 3.1, 2), and three models. It is evaluated on multilingual CrowS-Pairs (English, French, German, Dutch) and SeeGULL QA (English, French, Hindi), and reports $R(l)$ with bootstrap intervals over benchmark items.
The first falsifiable prediction is that under standard GPTQ and TACQ at 2–3 bits, $R(\text{hi})$ is significantly below $R(\text{en})$ for Llama-3.1-8B. We expect the gap to be smaller for Aya-Expanse-8B, mirroring the asymmetry already seen at full precision. If $R$ is near 1 everywhere, the measurement is still publishable as a negative result, and the method is unnecessary.
The second prediction is that fairness-aware saliency raises mean non-English $R$ at matched bits. It should cost no more than one MMLU point and no more than one F1 point on translated emrQA Medication. It should also show no added degradation on the distractor-laden Natural Questions check.
The third prediction is diagnostic. In models where retention fails, the per-language steering directions should drift apart after compression. If the directions stay aligned while retention still fails, the damage lies in language-specific pathways rather than the shared subspace. That would explain why English-only protection is insufficient.
References
- Debiasing Multilingual LLMs in Cross-lingual Latent Space
- UNLEARNING GEO-CULTURAL STEREOTYPES IN MULTILINGUAL LLMS
- Sustainable Modular Debiasing of Language Models
- Task-Circuit Quantization: Leveraging Knowledge Localization and Interpretability for Compression
- ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models
- Unsupervised Concept Vector Extraction for Bias Control in LLMs
- Investigating Bias in Multilingual Language Models: Cross-Lingual Transfer of Debiasing Techniques
- When Multilingual Models Compete with Monolingual Domain-Specific Models in Clinical Question Answering
- Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering