Which Sparse Features Survive Outlier-Suppressed Quantization?

Large language models develop a few "outlier dimensions": coordinates of the internal activation vector whose values are hundreds of times larger than the rest. They cause real problems when a model is compressed to 4-bit integers. The quantization grid has to stretch to cover the outlier, so small values elsewhere get rounded toward zero. Quantization methods fix this in two ways. Some shift and rescale channels and fold those changes into the weights. Others rotate the activation space so the outlier's energy is spread across every coordinate. Both are judged by perplexity and zero-shot accuracy. Interpretability researchers, meanwhile, read these same activations with sparse autoencoders (SAEs). An SAE is a learned dictionary that rewrites each activation as a sparse sum of roughly human-meaningful "features." Nobody has asked which of those features survive in the compressed model. The question matters because earlier evidence shows compression can break interpretable structure while standard metrics look healthy.

This proposal treats quantization as a structured noise source and asks which dictionary features that noise drowns. The core tool is a per-feature signal-to-noise ratio. For each feature, it divides the feature's typical signal strength by the quantization noise measured along that feature's reading direction. The ratio is computed from a few hundred calibration tokens and needs no retraining. We test whether this ratio predicts, feature by feature, which concepts a given quantizer destroys. We then use it to pick among, and tune, outlier-suppression methods so that concepts are preserved as well as perplexity.

Core Claim

Four lines of work motivate the proposal. Alignment Collapse Under KV Cache Quantization shows that quantizing the key–value cache silently erases refusal behavior. Mistral-7B lost 15.2% of refusals at a 1.03× perplexity increase, because the refusal direction lives in a low-energy subspace that outlier-stretched grids crush. That work studies one linear direction and explicitly leaves rotation-based quantizers unevaluated. When Pruning Meets Interpretability shows that a single aggregate perturbation energy governs how weight pruning damages SAE benchmark scores. The energy is $\mathrm{tr}(\Delta W \Sigma_x \Delta W^\top)$, where $\Delta W$ is the weight change and $\Sigma_x$ the input covariance. Under magnitude pruning, spurious-correlation removal degraded by up to 60% and targeted probe perturbation by up to 80% while KL divergence stayed above 0.96. That paper excludes quantization. SAEBench shows that reconstruction-style proxies mislead about downstream usefulness. On the Relationship Between Activation Outliers and Feature Death in Sparse Autoencoders links outlier severity to dead SAE features at training time, but never touches compression.

Our claim is sharper than "quantization hurts SAEs." Let $e_i$ be feature $i$'s encoder direction. Let $\eta$ be the difference between the quantized and full-precision residual-stream activations at the SAE's layer, and $\Sigma_\eta$ its covariance measured on calibration data. Then a feature's survival should be governed by

$$\rho_i = \frac{\mathbb{E}\left[(e_i^\top x)^2 \mid a_i(x) > 0\right]}{e_i^\top \Sigma_\eta\, e_i},$$

where $x$ is the full-precision activation and $a_i(x)$ is the feature's SAE activation. The numerator is the feature's signal strength when it fires. The denominator is the quantization noise along the direction the SAE reads.

The ratio makes concrete, testable predictions for each method. Naive round-to-nearest with a per-token scale inflates $\Sigma_\eta$ roughly uniformly, so every low-energy feature suffers. Rotation, as in QuaRot, shrinks the grid step but keeps the noise roughly isotropic. It should therefore help all features equally and still leave the weakest ones exposed. Channel shifting and scaling, as in Outlier Suppression+, makes $\Sigma_\eta$ anisotropic, so it helps some features and hurts others depending on how their encoders align with the rescaled channels. Features that read the outlier channel itself are a special case, tied to the "outlier-as-safety" regime from the alignment-collapse paper. Clipping should destroy them, while rotation should not.

We deliberately do not pursue the obvious framing: "does QuaRot's rotation destroy monosemanticity?" In full precision, an orthogonal rotation is exactly invertible. An SAE trained on the unrotated model can be applied to the rotated one by rotating its encoder and decoder, with zero change in output. So in full precision rotation is a no-op for interpretability. Only the interaction between rotation and rounding can matter, and that interaction is what $\rho_i$ measures.

Research Program

The work has three stages. The first two are a measurement study. The third is constructive, and we pursue it only if the first two show that QuaRot leaves a meaningful gap.

Stage one: build the audit. We take the publicly trained SAEBench dictionaries on Gemma-2-2B and Pythia-160M, which span ReLU, TopK, gated and Matryoshka architectures, and several layers. We quantize the host model under five settings: naive round-to-nearest at W8A8 and W4A4, Outlier Suppression+, QuaRot at W4A4 with 4-bit KV cache, and FP8 KV cache. We apply each fixed full-precision SAE, suitably rotated for QuaRot, to the quantized model's residual stream. For each feature we record three things: the ratio $\rho_i$ from 128 calibration sequences, the correlation between its full-precision and quantized activations on held-out Pile text, and its firing agreement (F1). We then run the full SAEBench suite: sparse probing, absorption, automated interpretability, spurious-correlation removal, targeted probe perturbation, unlearning on WMDP-bio, and RAVEL.

Stage two: separate destroyed information from relocated information. A fixed dictionary can fail for two different reasons. The quantized model may have moved a concept to a slightly different direction, or it may have erased the concept. We train fresh SAEs and linear probes directly on quantized activations. If fresh SAEs recover SAEBench scores, the damage is to dictionary transfer, and the remedy is cheap re-fitting. If they do not, the information is gone. This distinction is exactly what the pruning study left open when it evaluated only transferred SAEs.

Stage three: preserve concepts when choosing scales. If a gap remains, we replace Outlier Suppression+'s channel-scale objective, which minimizes output reconstruction error. The replacement maximizes a soft minimum of $\log \rho_i$ over the dictionary, or over a user-specified protected subset such as bias- or safety-related features. The scales are still folded into the weights, so inference cost is identical to Outlier Suppression+. The same objective can also choose, per layer, between rotating and scaling.

Baselines

The naive reference point is round-to-nearest quantization with per-token scales and no outlier handling. It sets the floor and should show the largest losses, concentrated in low-energy features.

The strongest existing method is QuaRot, which rotates weights, activations and the KV cache with randomized Hadamard transforms. QuaRot reports a 0.47 perplexity increase on WikiText-2 for Llama-2-70B at full 4-bit and roughly 99% retention of FP16 zero-shot accuracy. It is the strongest competitor because it removes coordinate outliers entirely and is the most likely to already preserve features. We run it at W4A4 with 4-bit KV cache on the same Gemma-2-2B and Pythia-160M checkpoints, the same calibration set, and the same SAEs as every other method. We found no reported SAEBench or feature-level result for QuaRot, or for any quantizer, on Lacuna. The comparison numbers will therefore be generated here, against the full-precision SAEBench scores of the same dictionaries.

Outlier Suppression+ is the migration-style competitor. It reports a 15.5% average GLUE gain over prior methods on 4-bit BERT-base. We run it with its published scale search under the same calibration budget.

A mixed-precision baseline keeps the few outlier channels in 16-bit and quantizes the rest. It tests whether the outlier channels alone explain the damage.

For the predictor itself, the comparisons are simpler per-feature statistics: activation frequency, mean activation magnitude, and encoder alignment with the outlier channels. We also include the aggregate perturbation energy from the pruning study, which cannot rank features at all. The key ablation replaces the measured $\Sigma_\eta$ with an isotropic noise model. It tests whether the direction of the noise, not just its size, carries predictive information.

Two results would count as success. The predictor succeeds if $\rho_i$ ranks per-feature degradation, by Spearman correlation, clearly above every one of these statistics on held-out layers. Stage three succeeds only if it matches QuaRot's perplexity within 0.1 on WikiText-2 while recovering more spurious-correlation-removal and targeted-probe scores than QuaRot does, beyond seed variance across SAE retrainings.

Why not just…?

Why not just check perplexity and KL divergence? The file offers three independent refutations. KL divergence above 0.96 coexisted with 60–80% intervention-metric collapse under pruning (When Pruning Meets Interpretability). Refusals flipped at a 1.03× perplexity increase (Alignment Collapse Under KV Cache Quantization). And SAEBench shows that loss recovered fails to predict disentanglement. We still report perplexity and KL for every setting, so the size of the gap between these metrics and the concept-level ones is itself a result.

Why not just use QuaRot and call it solved? This is the real threat. If rotation shrinks the grid step enough, every feature's $\rho_i$ may clear the threshold where degradation begins. Stage one settles this directly. If QuaRot keeps all SAEBench metrics within seed noise at W4A4 and 4-bit KV cache, we report that as the main finding and drop stage three. That would be a useful practical guarantee for anyone doing interpretability on deployed quantized models. Our prior against it is the isotropic-noise argument: rotation lowers noise uniformly and does not target the weakest features, which is where the refusal-direction losses appeared.

Why not just mean-center or zero out the outlier dimensions? Mean-centering fixes feature death during SAE training (On the Relationship Between Activation Outliers and Feature Death in Sparse Autoencoders). But it acts on the SAE's input, after rounding has already happened inside the network, so it cannot restore information the quantizer removed. Zeroing the outlier channels is known to restore similarity measurements (All Bark and No Bite). However, those channels act as attention no-ops (Quantizable Transformers) and, in some models, carry safety behavior. Our mixed-precision and clipping comparisons measure the cost of removing them directly.

Why not just retrain the SAE on the quantized model? Stage two runs exactly this experiment. Retraining is a legitimate fix when concepts have only moved. The audit's value is telling practitioners, before any retraining, whether they have moved or been erased.

Proposed Experiments

The first decisive check is whether $\rho_i$, computed from 128 calibration sequences, predicts which features lose firing agreement under each quantizer on held-out text. It must beat frequency, magnitude and outlier-alignment by Spearman correlation, and the isotropic-noise ablation must predict worse. That ablation is the test of whether the direction of the noise matters.

The second check is whether SAEBench's intervention metrics (spurious-correlation removal on Bias in Bios and Amazon Reviews, targeted probe perturbation, WMDP-bio unlearning) fall under round-to-nearest, Outlier Suppression+ and QuaRot at matched perplexity, and whether they fall in the order $\rho_i$ predicts. We expect features whose encoders read the outlier channel to be the specific casualties of clipping and scaling but not of rotation, which would bear on the dispute over whether massive activations carry meaning. SAEBench's roughly 65 minutes per SAE keeps a sweep over five quantizers, three layers and four dictionary architectures within a modest GPU budget.

The third check is the retrained-SAE comparison that separates relocated from destroyed concepts.

The fourth runs only if QuaRot leaves a gap. Scales chosen to maximize the feature signal-to-noise ratio must beat QuaRot on the intervention metrics without losing more than 0.1 WikiText-2 perplexity.

As a secondary validity check, we run the ConditionalFlip refusal audit from the alignment-collapse paper on Gemma-2 instruction-tuned checkpoints. We test whether a low $\rho$ along the refusal direction flags the same failures as that audit's per-channel diagnostic. This would connect the single-direction result to the full dictionary.