Outlier Preservation for Low-Bit Model Compression

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2305 papers457 conceptsfrozen 2026-10-04

When compressing a Large Language Model (LLM) or a Diffusion Transformer (DiT), the most intuitive approach is to treat all parameters equally. If you want to shrink a model to 4 bits, you might simply map the entire range of weights to 16 discrete levels. However, researchers have discovered that neural networks are not egalitarian. A tiny fraction of "outlier" values—often less than 1% of the total parameters—exerts a disproportionate influence on the model’s reasoning and stability.

Standard compression techniques fail because they face a "quantizer's dilemma": either they clip these high-magnitude outliers to fit the majority, destroying critical information, or they expand the quantization range to include them, which leaves the remaining 99% of weights with almost no numerical resolution. The emerging consensus across recent research is that the key to near-lossless compression lies in outlier preservation: identifying, isolating, and protecting these heavy hitters while aggressively compressing the rest.

The Statistical Signature of Importance

Outliers in LLMs are not random noise; they are structural. Lu Yin demonstrated in Outlier Weighed Layerwise Sparsity (OWL) that outlier density is a fundamental metric for parameter importance. By keeping more weights in layers with high activation outliers, OWL achieves massive improvements in perplexity at high sparsity levels.

To identify these critical values without expensive retraining, researchers use statistical proxies. Bowen Zhao and Qingqing Cao use kurtosis—a measure of the "tailedness" of a distribution—to flag weights that carry essential task-specific information in APT. Similarly, Alireza Ghaffari employs statistical shrinkage principles like the Adaptive LASSO in AdpQ to isolate outliers in a zero-shot, calibration-free manner.

Isolation Strategies: Quarantining the Heavy Hitters

Once identified, how do we protect these outliers without sacrificing hardware efficiency? Several distinct strategies have emerged:

1. Mixed-Precision Isolation The most direct method is to store outliers in high precision (FP16 or FP32) while the majority of the tensor is quantized to 3 or 4 bits. Ruslan Svirschevski introduced SpQR, which isolates less than 1% of sensitive weights. This approach allows 33B-parameter models to run on consumer GPUs with almost zero loss in accuracy. Amey Agrawal found in DynaQuant that protecting as few as 0.1% of weights is sufficient to enable aggressive compression of the remaining parameters.

2. Structural Reordering and Rotation Outliers often cluster in specific channels. Jung Hwan Heo and Se Jung Kwon proposed AdaDim, which switches quantization from per-output-channel to per-input-channel to "quarantine" these problematic activations. For hardware efficiency, Yilong Zhao and Zihao Ye developed Atom, which reorders outlier channels so they can be processed using high-throughput memory access patterns.

3. Mathematical Splitting Instead of mixed precision, some methods split the weight matrix itself. Fangzhen Lin and Jaewoo Song introduced SplitQuantV2, which uses k-means to partition weights into "quantization-friendly" clusters. By assigning outliers to their own sub-layers, the system prevents them from skewing the scaling factors for the rest of the model.

Beyond Inference: Training and Diffusion

The importance of outliers extends to the training phase and new architectures like Diffusion Transformers. Hui Cao showed in Metis that preserving "heavy hitter" singular values is more effective for training stability than mathematical transforms. By treating these outliers as essential structural components rather than noise, Metis enables robust 4-bit training (W4A4G4) for the first time.

In the realm of image generation, Lianwei Yang and Haokun Lin applied these concepts to DiTs in LRQ-DiT. They combined a "Twin-Log" approach for weights with an "Adaptive Rotation" scheme for activations, proving that managing outliers is just as critical for visual fidelity as it is for linguistic coherence.

Why It Works: The 4-bit Scaling Law

The move toward outlier-aware compression is validated by the "inference scaling laws" established by Tim Dettmers and Luke Zettlemoyer in The case for 4-bit precision. Their analysis of over 35,000 experiments revealed that 4-bit precision is the universal "sweet spot."

The reason 4-bit quantization works so well—and why sub-4-bit often fails—is directly tied to outliers. At 4 bits, there is just enough numerical "room" to represent the majority of the distribution while isolating the most extreme values. As we push toward 2 or 3 bits, the "quantizer's dilemma" becomes fatal unless we explicitly protect the outliers. As Seungcheol Park notes in his comprehensive survey, the shift from high-cost retraining to low-cost, outlier-aware calibration is what finally made the compression of 100B+ parameter models a practical reality.

Papers in this version

Showing 60 of 305 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 139 papers
  2. Version 1 · 2026-10-04 · 305 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_outlier_aware_quantization_a708cb1e,
  title = {outlier-aware quantization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/outlier-aware-quantization/rdr_a708cb1e345a4570bcb4559bb602b57f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_outlier_preservation_for_low_bit_model_c_a708cb1e_v1,
  title = {Outlier Preservation for Low-Bit Model Compression},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/outlier-preservation-for-low-bit-model-compression/rdr_a708cb1e345a4570bcb4559bb602b57f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}