Does Compression Make Edge LLMs Confidently Western? Culture-Stratified Calibration for Quantized Models

Phones and small boards like the Jetson cannot hold a full-precision language model, so on-device assistants run compressed copies. The usual method is quantization, which stores weights in 4 bits or fewer instead of 16. Compression is usually judged on English benchmarks, and so is calibration, meaning whether a model's stated confidence matches how often it is right. Both checks miss a population that edge deployment specifically serves: users outside the Western, English-dominated world that supplies most training data. For those users the model's knowledge is thinnest. We know that 4-bit quantization worsens calibration and leaves wrong answers at roughly 0.95 confidence (When Quantization Affects Confidence of Large Language Models?). We also know that models fall back to Western entities 30–40% of the time even when only an Asian one fits (Camellia). Nobody has asked what happens where these two effects meet.

Our claim is that quantization damage to calibration is not uniform across cultures, because it concentrates on items the full-precision model was already unsure about. Non-Western cultural knowledge is disproportionately such an item. We predict that compression turns correct-but-hesitant answers about non-Western entities into confident substitutions of a Western default, and that it does so faster as bit-width drops. That would degrade the model's ability to flag its own errors much more for non-Western queries than for Western ones. We will test this with a paired measurement design. We then test whether a tiny probe that reads the quantized model's own hidden states, trained on correctness labels drawn from several cultures, can restore error detection in a single forward pass. Throughout, we check honestly whether simpler fixes do just as well.

Core Claim

The measurable quantity is the culture gap in error detection. For a fixed model and bit-width, compute the area under the ROC curve (AUROC) for "confidence predicts correctness" on non-Western items. Compute the same on matched Western items. The gap is the difference between the two. We report the same difference in Expected Calibration Error (ECE), the average mismatch between confidence and accuracy across confidence bins.

The hypothesis has three parts. First, the gap is already present at 16 bits. Second, it widens monotonically from 16 to 4 to 3 bits, and the widening exceeds the seed-to-seed variation that quantization itself introduces. Third, the errors that open the gap are mostly confident Western-entity substitutions rather than random mistakes.

The mechanism comes from Proskurina et al.'s finding that quantization lowers confidence mainly on samples the full model was uncertain about, while overconfident errors survive untouched. Combined with the cultural thinness documented by DOSA and Camellia, this predicts that the hesitant-but-correct mass lost under compression is disproportionately non-Western. The losses get replaced by the model's high-prior Western answer.

If the gap does not widen with compression, the cross-cultural framing is unnecessary and generic quantization calibration fixes suffice. That outcome would be a useful negative result.

Prior Work and Its Roles

Three papers motivate the question. Proskurina et al. measure calibration under 4-bit GPTQ (a standard post-training quantization method) on six English commonsense benchmarks. They find calibration error rises by about 10% for LLaMA-7B on HellaSwag. Their study never stratifies by culture. Camellia and DOSA document the knowledge gap from the other side. Camellia reports a 10–20% extractive-QA accuracy gap between Asian and Western entities in native languages, against under 5% in English. DOSA reports GPT-4 accuracy of 22% on Chhattisgarh artifacts against 75% on Kerala. Neither measures confidence or compression.

Two papers constrain the design. On the Impact of Calibration Data in Post-training Quantization and Pruning shows that the 128 sequences used to set quantization scales can swing downstream accuracy by nearly 10% under pruning, though only about 0.2% under SpQR. Any culture-gap claim must therefore be measured across several such draws, not one. Their calibration sets also came only from English web corpora, so their composition is itself a candidate cause we must test.

CoCoA trains cultural LoRA adapters (small low-rank weight updates) that cut a Cultural Bias Score from 43 to 24. Its "Neutral Calibration Loss" balances entity preferences, not probabilities. It was run only at 16 bits. CoCoA therefore addresses which entity the model picks, while we address whether the model knows when its pick is wrong.

Two papers supply methods. Uncertainty Calibration for Tool-Using Language Agents trains a small multilayer perceptron on frozen hidden states using 500 questions and drops ECE on TabMWP from 0.690 to 0.035. It has only been tested on math tools at full precision. EdgeLoRA shows that reading base-model hidden states to route requests costs almost nothing on Jetson and Raspberry Pi hardware. We reuse its integration pattern for a confidence head instead of a router.

Large Language Models Must Be Taught to Know What They Don't Know is both a method and a hypothesis we test. It shows that LoRA calibration tuning on about 1,000 graded examples cuts open-ended ECE from ~30% to 10%, and that this skill transfers across academic subjects. Its domains are all MMLU subjects, and its scoring needs a second pass. We ask whether the transfer crosses cultural boundaries, and whether it survives a quantized base model.

Research Program

The first component is a measurement study.

Models and quantization settings. We use Llama-3.2-3B, Llama-3.1-8B and Qwen3-8B, the sizes EdgeLoRA and CoCoA already run. Each is quantized at 16-bit, 4-bit GPTQ, 3-bit GPTQ, SpQR, and APTQ's attention-aware mixed precision at a 3.8-bit average (APTQ). Every quantized setting is repeated over five random draws of quantization calibration data.

Labeled data with built-in controls. Correctness-labeled items come from three sources:

  • Camellia's extractive QA. Its paired Asian-entity and Western-entity contexts in the same nine languages give a built-in matched control.
  • DOSA artifact identification across 19 Indian states.
  • CAMeL entity tasks for Arabic.

The Western reference is the Western-entity half of Camellia plus the ARC, BoolQ and HellaSwag items Proskurina et al. used. Using the same items lets their numbers anchor ours.

Confidence and error analysis. Confidence is the length-normalized answer probability. For every error we record whether the wrong answer is a Western entity, which yields a "confident Western substitution" rate: errors that are Western picks at confidence above 0.9.

The second component is the intervention. We train a two-layer probe on the quantized model's own mid-layer hidden states, pooled over the answer span. Labels are that quantized model's correctness, because which items are right changes after quantization. The probe adds one small matrix multiply per answer. It runs in the same forward pass as generation and is deployed as a side head in an EdgeLoRA-style runtime. Its output drives abstention, or deferral to a cloud model.

The key design choice is leave-cultures-out training. We train on some cultures, for example Chinese, Japanese and Korean Camellia items, and test on cultures never seen, for example DOSA's Indian states and CAMeL's Arabic items. This tells us whether the probe has learned a culture-general signature of "the model is falling back to its default", or merely memorized per-culture difficulty. We use about 1,000 training items, the data budget of Kapoor et al.

Baselines

Naive reference. The quantized model's raw answer probability, with no correction. Proskurina et al. report that this signal leaves wrong answers near 0.95 confidence after 4-bit GPTQ. Everything is measured against it, and against the same model at 16 bits.

Post-hoc temperature scaling. We fit one temperature per model and bit-width on held-out data. In a stronger variant, a separate temperature is fitted per language, using language identity as a cheap stand-in for culture. Both are fitted on the same 1,000 items the probe sees.

Kapoor et al.'s LoRA calibration tuning. This is the strongest existing method for open-ended calibration. Its reported result is open-ended MMLU ECE falling from about 30% to 10%. We run it two ways: trained on 16-bit and then quantized, and trained directly on the quantized base. Each variant gets the same 1,000 items, the same graded labels, and the same hyperparameter search budget as the probe (20 configurations). We also report its extra latency on Jetson AGX Orin, since it requires a second pass.

Hidden-state probe trained only on Western items. This applies Liu et al.'s probe recipe, which reports ECE 0.690 to 0.035 on TabMWP, to Western-only data of the same size. It is the ablation that isolates our key ingredient: if it matches the culture-stratified probe on unseen non-Western cultures, then cultural diversity in the training labels does not matter.

Culturally mixed quantization calibration data. We rebuild the 128-sequence quantization set from native-language text drawn from the evaluated cultures, with no probe at all. This is a pure data fix.

What counts as winning. No reported result was found on Lacuna for culture-stratified calibration of quantized models, so absolute targets are ours to set. The probe beats a baseline if it does at least one of the following at 3–4 bits on held-out cultures:

  • shrinks the culture AUROC gap by at least half relative to that baseline;
  • lowers selective error at 80% coverage on non-Western items by more than three standard deviations of the PTQ-seed variation.

In either case it must not worsen Western-item ECE, and it must add under 2% to token latency on Jetson.

Why not just…?

Why not just use temperature scaling? A single temperature changes how sharp confidences are but never their order. It can lower ECE but cannot change AUROC, so it cannot separate a confident wrong Western substitution from a confident right answer when both sit near 0.95. Per-language temperatures are a real threat for ECE, and we expect them to recover much of the calibration gap. The experiment settles whether they also recover error detection; if the culture gap is mainly about ranking, they will not. If per-language temperatures close both gaps, we will report that as the recommended fix.

Why not just mix culturally diverse text into the quantization calibration set? This is the most dangerous alternative, because it is free at deployment time. Williams and Aletras show that calibration-set composition matters a lot under pruning, though much less under SpQR. If the culture gap is caused by quantization scales tuned to English activations, this fix should close it, and we will say so. If the gap comes from the Western default already present at 16 bits and merely amplified by compression, rebalancing the scales will not restore knowledge the model never had. The full-precision gap measurement tells these two cases apart. We will also stack the fix with the probe to see whether the two are complementary.

Why not just apply Kapoor et al.'s calibration LoRA, which already transfers across domains? Its transfer evidence covers MMLU subjects only. Its two-pass scoring roughly doubles memory traffic on bandwidth-bound edge chips. Whether it survives a 3-bit base model is untested. We run it in both orders (tune then quantize, quantize then tune) and with the same training cultures as the probe. If it matches the probe on held-out cultures, the remaining argument for the probe is latency alone, which we measure rather than assume.

Why not just ask the model how confident it is, or prompt it with a local persona? Kapoor et al. show that verbal confidence shifts with prompt wording while accuracy does not. DOSA finds that a second guess after negative feedback gives negligible gains, pointing to missing knowledge rather than prompt misunderstanding. We include verbal confidence as a cheap reference row and do not expect it to compete.

Decisive Checks

Check one: is the gap real? Plot the culture AUROC gap against bit-width, with error bars from the five quantization seeds. The core claim survives only if the gap at 3 bits exceeds the 16-bit gap by more than the seed spread. It must also hold on the paired Camellia contexts, where language and task format are identical and only entity origin differs.

Check two: what causes the widening? The confident-Western-substitution rate should rise with compression. If errors under compression are instead random, the Western-default mechanism is wrong even if the gap exists.

Check three: does a cheaper fix make the probe unnecessary? On held-out cultures, the culture-stratified probe must beat both the Western-only probe and the culturally mixed quantization data. If either matches it, we drop the probe and report the cheaper fix.

Check four: does it work on the device? Run the winning configuration on Jetson AGX Orin and Raspberry Pi. Report added latency, memory, and selective accuracy when low-confidence non-Western queries are deferred.