Does Variance-Based Compression Make Language Models Hallucinate the Popular Answer?

Principal component analysis (PCA), a method that finds the few directions along which a set of vectors varies most, is now used to make large language models cheaper to run. Recent methods rotate each layer's internal activations into their principal-component frame and discard the low-variance coordinates, which buys large inference speedups at almost no cost in perplexity. Perplexity, though, is an average over common text. Facts about rare entities are, almost by definition, low-frequency signals. We propose to test whether variance-based compression removes exactly those signals. If it does, a compressed model would answer tail questions with the answer a popular entity would have, which is a form of bias amplification: the compressed model concentrates its output further on the majority modes it already favored.

The claim has two parts. First, PCA-sparsified models show a larger accuracy loss on low-popularity entities than on popular ones, and their errors are disproportionately substitutions of more popular answers. In ambiguous social questions, they fall back on stereotypes more often. Neither effect would show up in perplexity or in standard reasoning benchmarks. Second, the cause is the calibration covariance, the activation statistics from which the principal directions are computed. Reweighting that covariance toward rare-entity text should recover most of the tail accuracy at the same sparsity and speed. This is borrowed directly from the importance-reweighting remedies used against bias amplification in generative models.

Why this gap, and why now

When Not to Trust Language Models motivates the concern. It introduced PopQA, 14,000 Wikidata questions indexed by the subject's Wikipedia page views. It found that parametric recall falls steeply with popularity: on the 4,000 least popular entities, GPT-j 6B reaches 16% and GPT-3 davinci-003 reaches 19%. Tail knowledge is already fragile, so it is where damage from compression would show first.

La RoSA supplies the method we stress-test. It computes layerwise PCA rotations on WikiText2 and zeroes the lowest-variance rotated coordinates. It reports a 1.30x wall-clock speedup at 40% activation sparsity on LLaMA2-7B with a 0.17 perplexity gap, and zero-shot accuracy 1.77% and 17.14% better than the magnitude-pruning methods TEAL and CATS. It evaluates perplexity, ARC, MMLU, GSM8K and MATH500, but never popularity-stratified factual recall. The file contains no La RoSA result on PopQA, TruthfulQA or BBQ.

The same blind spot applies to the related compressions in Loki, which approximates attention with PCA-projected keys, and SP^3, which prunes the hidden dimension through PCA projection.

Effective Dimensionality Reduction for Word Embeddings gives the mechanistic hint. In static embeddings, the mean vector and top few principal components encode frequency-driven, non-discriminative structure; removing them improves semantics. If contextual activations behave similarly, the top components that compression keeps are frequency directions. Rare-entity identity would then live below the cutoff.

The bias-amplification literature frames what we expect to see and supplies the remedy. FICS shows a conditional GAN reducing blond males from 5.85% of the training data to 3.97% of generations. The iterated-learning analysis shows posteriors collapsing onto the strongest prior. Reference-based reweighting (Choi et al., Kim et al.) counters such skews by reweighting the statistics a model is fit to. We apply that idea to the covariance matrix that defines the compression, not to generator training.

The closest existing work is Spectral Editing of Activations (SEA). SEA builds cross-covariances from 25 curated truthful/hallucinated or biased/unbiased pairs and projects out undesirable directions. It raises LLaMA-2-7B TruthfulQA MC1 from 36.96 to 39.41 and BBQ accuracy from 43.02% to 56.17%. SEA edits behavior deliberately using paired supervision. We ask a different question: whether an unsupervised, efficiency-motivated spectral truncation causes majority-mode hallucination as a side effect. Our fix needs no paired data, only entity-frequency estimates.

CASAL finds that known and unknown queries are linearly separable in hidden states. We use this as a probe target: does that separation survive truncation?

Research Program

The first stage is a spectral diagnosis on LLaMA2-7B, the model La RoSA was tuned for, with a second model family for replication. At each layer we compute the principal components on WikiText2 exactly as La RoSA does. We then pass PopQA questions through the uncompressed model and decompose their activations onto those components.

We ask two questions of this decomposition. First, where does entity-specific information live? We train linear probes for subject identity, for CASAL's known/unknown label, and for the correct object entity, restricting each probe to the top-$k$ components versus the discarded ones. Second, does the discarded share of the information grow as popularity falls? We measure this across PopQA's popularity deciles. If tail entities carry a larger fraction of their probe-decodable signal in low-variance components, the mechanism is confirmed before any generation is run.

The second stage measures behavioral amplification. We sparsify at 20%, 40% and 60% and evaluate three things.

The first is PopQA accuracy per popularity decile. We summarize it as the tail gap: the accuracy drop on the bottom 4,000 questions minus the drop on the top decile.

The second is a substitution rate. It is the fraction of wrong answers that match a valid object for the same relation belonging to a more popular entity, with popularity measured by the page views the benchmark already provides. In the style of FICS, we report the share of popular-entity answers in the compressed model's outputs divided by that share in the uncompressed model's outputs. A ratio above one is amplification.

The third is BBQ, split into ambiguous and disambiguated contexts. BBQ reports that UnifiedQA's errors in ambiguous contexts are stereotype-aligned roughly 77% of the time, and that models are 3.4 points less accurate when facts contradict stereotypes. We test whether compression raises either number. TruthfulQA MC1 serves as a third probe of imitative-falsehood priors.

The third stage is the remedy: frequency-reweighted calibration. We replace the unweighted calibration covariance with

$$C_w = \sum_i w_i \, h_i h_i^\top \Big/ \sum_i w_i,$$

where $h_i$ is a calibration activation, $w_i \propto p(e_i)^{-\alpha}$, $p(e_i)$ is the estimated corpus frequency of the entity mentioned in the calibration passage, and $\alpha \in [0,1]$ controls how strongly rare entities are upweighted. At $\alpha = 0$ this reduces to La RoSA. The calibration text is Wikipedia passages spanning the popularity range, disjoint from PopQA subjects so that test facts are not leaked. Rotation, Top-K sparsity and the speed accounting are unchanged, so any gain comes from which directions are kept.

Baselines

The naive reference point is the uncompressed model. It defines each model's own popularity curve, and every amplification number is computed relative to it.

The strongest existing method is La RoSA as published: WikiText2 calibration and the same sparsity levels, reproduced with its code settings. Its reported strength is the 0.17 perplexity gap at 40% sparsity. On popularity-stratified factual QA, there is no reported result found on Lacuna for La RoSA, so we establish that baseline ourselves.

The remedy beats La RoSA if, at equal sparsity and measured wall-clock speed, it closes at least half of La RoSA's tail gap and brings the substitution ratio significantly closer to one. It must do so while holding perplexity within La RoSA's gap and zero-shot ARC and MMLU within one point. Significance is assessed with bootstrap intervals over questions.

Magnitude pruning (TEAL/CATS style, which the La RoSA paper compares against) is a second baseline. It uses no variance ordering, so it separates popularity damage caused by variance ranking from damage caused by sparsity itself.

The key ablation uses the same popularity-spanning Wikipedia calibration set with $\alpha = 0$. It separates the effect of calibration content from the effect of the reweighting.

A random-rotation control at the same sparsity tests whether PCA ordering specifically is responsible.

Loki's low-rank keys are an optional second compression family. We run them only if the activation results are positive.

All methods share the same calibration token budget, the same evaluation prompts and greedy decoding. Each method gets an equal hyperparameter search of about ten settings, tuned on a held-out PopQA split, never the test split.

Why not just…?

Why not just calibrate on more diverse data? This is the most serious threat. The unweighted diverse-calibration ablation tests it directly. If that ablation matches the reweighted version, we will report that the fix is data content rather than weighting, and the diagnosis and the amplification measurements remain the contribution. We expect weighting to matter because frequent entities still dominate any natural corpus's covariance, and the popularity skew in PopQA recall suggests that skew is steep. That expectation is exactly what the ablation checks.

Why not just use less sparsity? Trading speed for tail accuracy is always possible. Reporting full tail-gap-versus-sparsity curves makes that trade explicit. The claim is that reweighting moves the whole curve, not one operating point. If the curves coincide, the remedy fails and we will say so.

Why not just use magnitude pruning, which does not rank by variance? La RoSA reports magnitude methods 1.77 to 17.14 points worse on zero-shot accuracy, so they are not a free alternative. They could still be less popularity-biased. We will learn that from the magnitude-pruning baseline, and the result would be informative either way.

Why not just patch the tail with retrieval? Adaptive retrieval lifts GPT-3 davinci-003 to 46.5% on PopQA, but it hurts in about 10% of cases through retrieval corruption, and it adds latency that erodes the purpose of compression. We will run adaptive retrieval on top of both La RoSA and the reweighted model. If retrieval fully erases the difference, the remedy matters only for retrieval-free deployments, and we will state that scope honestly.

Decisive Checks

The central test comes before any fix. If probes show no popularity-dependent shift of entity information into low-variance components, and La RoSA's tail gap and substitution ratio sit within noise of the uncompressed model, the hypothesis is false. That negative result would still usefully certify PCA sparsification for factual use.

If amplification appears, the second decisive test is whether it tracks the spectral diagnosis layer by layer. Layers where tail information sits below the cutoff should be the layers whose selective uncompression restores the most tail accuracy.

The final test is whether reweighting closes the gap beyond the diverse-data ablation. The BBQ ambiguous-context stereotype rate then shows whether the same mechanism links factual hallucination to social majority priors, as the bias-hallucination view in Addressing Bias and Hallucination in Large Language Models predicts but never measures.