Keep the Text, Drop the Label: Label-Free Calibration for Dialect-Safe 4-bit Codebooks

Large language models are routinely shrunk to 4 bits per weight so they fit on cheap hardware. The best current methods do this by learning a small table of 16 allowed values, called a codebook, for each row or layer. They fit the table with a weighted clustering step, where each weight's error counts in proportion to how strongly that weight's input channel fires on a handful of "calibration" sentences. Those calibration sets are tiny. AAAC uses four C4 sequences, and any4 uses a single 88-word prompt. Weights that matter mainly for regional varieties of a language, such as Tunisian Arabic, Argentine Spanish, or Kiswahili spoken questions, therefore barely fire during calibration and get coarse treatment. The obvious fix is to add "dialect data" to the calibration set. In practice, dialect data at scale is labeled by geography: geotags, country web domains, broadcast location. Sociolinguistic work shows these labels are badly wrong. Geolocated Tunisian tweets were only 49.3% correct against human validation in NADI 2024, and country-domain web text is full of syndicated standard-variety content.

We propose to stop trusting the labels altogether. Instead of choosing calibration text by where it came from, we choose it by what quantization actually breaks. We build a provisional codebook, run the full-precision and quantized models side by side over a large unlabeled pool of text, and upweight the tokens whose output distributions diverge most. Those weights feed back into the activation statistics that drive the clustering. Geographically scraped text can stay in the pool as raw material, but its geographic label is never used. The claim is that this recovers most of the dialect-specific damage that human-validated dialect calibration would recover, at no inference-time cost. It should also clearly beat calibration on geography-labeled "dialect" data, because a large share of that data is not dialectal at all.

Core Claim

Activation-weighted codebook methods compute one saliency number per input channel, an average of squared activations over calibration tokens, and minimize saliency-weighted reconstruction error. We make two claims. The first is diagnostic: dialect-specific degradation after codebook quantization is predicted by the mismatch between channel second moments on dialect text and on the calibration text. Geography-labeled calibration data narrows this mismatch only in proportion to how often its labels are right. The second is constructive: replacing the uniform token average with a divergence-weighted average closes most of the gap without any variety labels. Here a token's weight is a capped function of how much the provisional quantized model's next-token distribution departs from the full-precision model's. Concretely, for input channel $j$,

$$s_j = \frac{\sum_t w_t\, x_{t,j}^2}{\sum_t w_t}, \qquad w_t = \min\!\left(\exp\!\big(\mathrm{KL}_t/\tau\big),\, w_{\max}\right),$$

where $x_{t,j}$ is the activation of channel $j$ at pool token $t$, $\mathrm{KL}_t$ is the divergence between full-precision and provisional-quantized next-token distributions at $t$, $\tau$ is a temperature, and $w_{\max}$ caps any single token's influence. Codebooks are then refit by minimizing $\sum_j s_j (W_{ij} - c_{k(ij)})^2$, where $c_{k(ij)}$ is the codebook entry assigned to weight $W_{ij}$, and the loop repeats two or three times.

This is a group-agnostic, worst-case-leaning reweighting, similar in spirit to methods that upweight a model's own errors. It acts on calibration statistics rather than on training, so the output is still a static codebook that runs on existing kernels.

Relation to Prior Work

The method comes from activation-weighted clustering. And the Bit Goes Down introduced weighting vector-quantization error by input activations. any4 learns a per-row 16-entry lookup table with activation-weighted k-means. AAAC learns two per-layer codebooks from forward-pass statistics in minutes. NoWag weights k-means by the activation second moment. All four treat the calibration average as given. We keep their clustering unchanged and change only which tokens the average is taken over and how they are weighted. The any4 authors note they never checked whether single-prompt calibration biases outputs; that unchecked question is our starting point.

Two compression-fairness studies motivate the work. Understanding the Unfairness in Network Quantization traces disparate quantization damage to subgroup gradient norms and Hessian traces. It also reports that quantization-aware training widens the gap, from 3.3% to 25.4% on UTK-Face compared with 18.8% for post-training quantization. That study is vision-only and needs group labels. Unmasking the Hidden Fairness, Bias, and Safety Costs of Compression finds that calibration composition moves bias non-monotonically. It also identifies a "fairness mirage": apparent bias reduction that is really uniform capability collapse. That finding constrains our evaluation. We measure damage relative to the full-precision model per variety, never absolute parity.

TTQ attacks calibration domain shift by recomputing uniform-grid parameters per prompt at inference. We instead fix the codebook offline, which keeps batching simple.

The proxy-mismatch literature supplies both the warning and the evaluation data. NADI 2020, Bergman et al., Pavalanathan et al., the DSL 2014 report, and Sun et al. document that residence, opt-in geotagging, publication site, and country domain each misassign variety. NADI 2024 and DSL-TL show that many sentences are valid in several varieties or in none. That overlap is another reason a label-free criterion should be preferred to hard geographic bins. DAFAIR also mitigates bias without demographic labels, but it does so by regularizing a full-precision encoder during fine-tuning and does not touch compression.

Research Program

The work has three parts.

The first part is diagnosis. We quantize Llama 3 8B and the Arabic-centric Jais with AAAC-style and any4-style codebooks under their published calibration recipes. We then measure, per variety, the increase in per-token negative log-likelihood and in KL to the full-precision model. For each held-out sentence we compute the second-moment ratio between its activations and the calibration activations on the most salient channels. We test whether this ratio predicts that sentence's damage better than sentence length, perplexity under the full-precision model, or the dialectness score of NADI 2024. The dialectness score is a continuous rating of how far a sentence is from Modern Standard Arabic.

The second part is a proxy-versus-truth calibration study. We build token-matched calibration sets three ways. One uses geography-labeled text: geotagged tweets and country-domain web text for each target variety. One uses human-validated text: NADI's validated development data for Arabic and DSL-TL training data for Portuguese, Spanish, and English. The last is standard C4. We then ask how much of the human-validated improvement the geography-labeled sets recover. We also ask whether recovery tracks the measured label accuracy for each country.

The third part is the divergence-weighted method itself. The pool combines C4, the geography-labeled text with labels stripped, and multilingual web text, and contains no evaluation sentences. We study the method's sensitivity to $\tau$, $w_{\max}$, pool size, and number of outer iterations. We also check what the upweighted tokens are. Using held-out human labels only for analysis, we measure what fraction are validated dialect text and what fraction are junk such as code, boilerplate, or broken Unicode.

Baselines

The naive reference point is the full-precision model alongside plain round-to-nearest INT4. Together these bound the problem. Every damage number is reported as the per-variety gap to full precision, and accuracy is reported next to it so that a collapse cannot pass as fairness.

The strongest existing method is AAAC with its published calibration of four 2,048-token C4 sequences. AAAC reports recovering about 60% of the WikiText-2 perplexity gap between 4-bit and FP16, against about 28% for AWQ. It also reports 64.21% average accuracy on an 8-task suite against 64.59% for full precision. We also run any4 with its 88-word prompt. That paper reports C4 perplexity of 9.40 on Llama 3 8B, against 9.52 for NF4 and 9.89 for INT4. We reproduce these headline numbers before comparing. No reported result was found on Lacuna for any codebook method on dialectal or variety-specific evaluation, so the dialect baselines are numbers we produce.

For fairness, every calibrated method receives the same unlabeled pool. The pool-mean variant of each baseline computes saliency as a uniform average over the whole pool, so differences cannot come from data access. Each method gets the same per-method tuning budget on a validation split that is disjoint from test. Bit-width, group size, and codebook overhead are held fixed. We also run AWQ and TTQ as uniform-grid references. TTQ is the strongest claim that per-prompt adaptation makes calibration irrelevant. On OPT-350M at 3 bits it reports perplexity of 25.3 against 25.02 for AWQ.

Three calibration baselines isolate the key ingredient. The first uses geography-labeled dialect calibration, which is the obvious fix. The second uses human-validated, variety-balanced calibration, which is an oracle that requires labels. The third is a shuffled-weight control. It applies the same distribution of token weights as our method, but assigns them to random tokens. This separates "reweighting by divergence" from "any heavy-tailed reweighting."

We beat the strongest baseline if two things hold. First, on the worst-off variety, the gap to full precision falls by more than the seed-to-seed spread relative to pool-mean AAAC. Second, standard WikiText-2/C4 perplexity and 8-task accuracy stay within that same spread of AAAC. We would call the method a clear success if it recovers most of the oracle's improvement while also beating the geography-labeled set.

Why not just…?

Why not just add geotagged or country-domain dialect text to the calibration set? This is the real threat, and the second part of the program is designed to settle it. The evidence against it is indirect but strong. Geolocated Tunisian labels were 49.3% accurate in NADI 2024. Opt-in geotaggers skew toward particular age and gender sociolects according to Pavalanathan et al.. Country-domain text carries syndicated standard content, per Sun et al. and the DSL 2014 report. Much of a "Tunisian" calibration set may therefore light up the same channels as standard text. The threat could still win. If geography-labeled calibration matches the human-validated oracle, the proxy problem does not matter for calibration. We would report that, and our method's remaining value would rest only on not needing any language-specific collection.

Why not just use test-time quantization so calibration never matters? TTQ adapts uniform scales per prompt. It was evaluated only on clean benchmark text, does not learn non-uniform codebooks, and adds per-prompt profiling that complicates batching. We run it on the same dialect sets, including SD-QA's noisy speech transcripts, and report throughput alongside damage. If it matches our dialect results at acceptable speed, it is the better practical answer for uniform grids. That would still leave codebook methods, which any4 shows beat uniform grids on perplexity, without a fix.

Why not just calibrate on a much larger, more diverse random sample? any4 found that more C4 tokens did not beat its 88-word prompt on perplexity. That suggests uniform averaging over more majority text is not the lever. Our pool-mean baseline uses exactly the same large pool with uniform weights, so this experiment directly measures how much of any gain comes from data volume and how much from the divergence weighting.

Why not just fine-tune with quantization-aware training? Liu et al. report that it widens subgroup gaps more than post-training quantization. It also costs far more compute than forward-only codebook fitting. We do not treat it as a primary baseline. We include it only if compute allows, as a reference point on one model.

Proposed Experiments and Decisive Checks

Arabic evaluation uses three NADI 2024 tasks. The first is dialect-to-MSA translation, where the reported winning system, Jais with a one-shot prompt, reached 20.44 BLEU. We run the full-precision Jais on the same task and measure the drop under each quantizer. The second is per-country log-likelihood on the human-validated multi-label test tweets. The third is a stratified analysis by dialectness score.

Variety evaluation uses DSL-TL. We measure per-variety log-likelihood and prompted three-way identification of variety A, variety B, or "both or neither." The reported 9-class macro F1 of 0.540 for fine-tuned mBERT/XLM-R serves as context, not a target. SADSLyC adds per-region log-likelihood, where Shamali at 3.24% and Shargawi at 4.91% of the corpus are the low-resource stress cases. Spoken interaction uses SD-QA question answering on both gold and ASR transcripts across its 24 dialects. Standard-quality checks are WikiText-2 and C4 perplexity and the 8-task suite.

Three outcomes would decide the question. The first is that the second-moment mismatch predicts per-sentence damage on held-out dialect text. If it does not, the saliency story is wrong and the method's motivation goes with it. The second is that geography-labeled calibration recovers measurably less of the oracle's dialect improvement than human-validated calibration does. If not, the proxy-mismatch angle does not apply here. The third is that the divergence-weighted codebook closes most of the gap between pool-mean AAAC and the oracle on the worst variety. It must do so without hurting standard perplexity and while beating the shuffled-weight control. Failing that control would mean any heavy-tailed reweighting works and divergence carries no information.

Throughout, every reduction in a cross-variety gap is reported next to the per-variety drop from full precision, to rule out the fairness mirage. Every gain must hold across three calibration seeds.