Orthogonal Rotations for Outlier Smoothing in Quantized Models

472 concepts201 papers2 proposals

Quantizing Large Language Models (LLMs) to 4-bit or 2-bit precision is often described as the "holy grail" of efficient AI, promising to slash memory usage by 75% or more. However, as Liu et al. note in their work on SpinQuant, the primary obstacle to this goal is the presence of "outliers"—extreme numerical values in weights and activations that ruin quantization accuracy.

A new research direction, often termed incoherence processing, has emerged to solve this by using orthogonal transformations to "smooth out" these outliers before they are compressed.

The Outlier Problem

In standard uniform quantization, we map a range of floating-point numbers to a small set of discrete bins (like the 16 levels available in 4-bit integers). If a distribution is "well-behaved" (like a Gaussian), the bins can be spaced efficiently to capture most of the information.

However, LLMs exhibit heavy-tailed distributions with massive outliers. As Alistarh et al. explain in QuaRot, these outliers force the quantization grid to expand its range significantly. This expansion makes the bins so wide that the majority of standard values—the "signal"—fall into the same few bins, leading to massive information loss.

Smoothing via Rotation

The core insight of this research area is that while outliers are extreme in specific dimensions, they do not necessarily represent high energy across the entire vector space. By applying an orthogonal transformation (a rotation in high-dimensional space), we can redistribute the magnitude of these outliers across all dimensions.

Chee et al. pioneered this in QuIP, arguing that "incoherence"—ensuring that weights and their importance are spread evenly—makes the rounding process robust. Mathematically, if we have a vector with one huge spike (e.g., $[10, 0, 0, 0]$), rotating it might result in a vector where the energy is shared (e.g., $[5, 5, 5, 5]$). The total energy (the norm) remains the same, but the peak-to-average ratio, or "crest factor," is drastically reduced.

Mechanisms of Transformation

Researchers have proposed several ways to implement these rotations:

  • Random Hadamard Transforms: Panferov et al. in HIGGS and Xi et al. in Training Transformers with 4-bit Integers utilize the Walsh-Hadamard Transform (WHT). The WHT is a structured orthogonal matrix that can be computed very quickly. It effectively "smears" outlier magnitudes, forcing the distribution toward a near-perfect Gaussian shape that is much easier to quantize.
  • Learnable Rotations: Rather than using fixed random rotations, Hu et al. propose OSTQuant, which learns orthogonal and scaling transformations to maximize "Quantization Space Utilization Rate." Similarly, Liu et al. optimize rotation matrices via the Cayley transform to specifically target the kurtosis of the data.
  • Dual Transformations: Some methods, like DuQuant by Lin et al., combine rotations with permutations (like "zigzag" reordering) to balance the outlier load across different blocks of the model.

Maintaining Inference Efficiency

A critical requirement for these methods is that they must not slow down the model during actual use. The authors of FPTQuant, van Breugel et al., demonstrate that these transformations can often be "fused" directly into the model's existing weights.

Because linear layers are mathematically associative, applying a rotation $R$ to the activations and its inverse $R^T$ to the weights results in the same output ($ (xR)(R^T W) = xW $). By pre-computing these rotations and folding them into the weights during the quantization process, the model achieves the benefits of a smooth distribution with zero additional computational cost at runtime.

As Chen and Ma show in their comprehensive study, this smoothing process is so effective that it allows simple integer (INT) formats to outperform more complex floating-point (FP) formats in 4-bit regimes, paving the way for highly efficient, hardware-friendly LLM deployment.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 186 papers
  2. Version 1 · 2026-10-04 · 239 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_incoherence_processing_8e6a53a8,
  title = {incoherence processing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/incoherence-processing/rdr_8e6a53a838254fc4a08cec14c4d9c8f5},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_incoherence_processing_8e6a53a8_v2,
  title = {incoherence processing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/incoherence-processing/rdr_8e6a53a838254fc4a08cec14c4d9c8f5/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake