Incoherence Processing for Low-Bit Quantization

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2239 papers387 conceptsfrozen 2026-10-04

Large Language Models (LLMs) and Diffusion Transformers (DiTs) are notoriously difficult to compress. When we attempt to squeeze their 16-bit weights and activations into 4-bit or 2-bit integers, performance often collapses. The culprit is not the average value, but the "massive outlier"—rare, high-magnitude spikes in specific channels that dominate the numerical range.

A rapidly growing body of research has converged on a powerful solution: Incoherence Processing. By applying orthogonal transformations (like the Hadamard transform) to weights and activations, researchers can "smear" these outliers across many dimensions, turning a "spiky" distribution into a smooth, Gaussian-like one that is far easier to quantize.

The Tyranny of the Outlier

To understand why outliers break quantization, imagine trying to represent a set of numbers using only 16 levels (4 bits). If your numbers are all between 0 and 1, your levels are spaced 0.06 apart. But if a single outlier appears at 100, your levels must now span from 0 to 100, meaning each level is 6.25 apart. Suddenly, every value between 0 and 1 is rounded to zero. The signal is lost to the scale.

In Transformers, these outliers are not random noise; they are structural. As Haocheng Xi and colleagues note in Training Transformers with 4-bit Integers, these spikes force a coarse numerical scale that obscures the subtle information needed for model accuracy.

Smoothing via Orthogonal Rotation

The core intuition of incoherence processing is that we can change the "view" of our data without changing its fundamental properties. In linear algebra, an orthogonal transformation (like a rotation) preserves the lengths of vectors and the angles between them.

If we have a vector with one massive spike (e.g., $[0, 100, 0, 0]$), rotating it into a different coordinate system can redistribute that energy. After a Walsh-Hadamard Transform (WHT), that same vector might look like $[50, 50, -50, -50]$. The "energy" is the same, but the maximum value has dropped from 100 to 50.

Jerry Chee and the authors of QuIP formalize this as "incoherence." By applying random orthogonal transformations, they ensure that weights and their importance are spread evenly across dimensions, making 2-bit quantization viable for the first time.

The Invariance Trick

The beauty of this approach is that it can be implemented with zero change to the model's mathematical output. For any linear layer $Y = WX$, we can insert an orthogonal matrix $R$ and its transpose $R^T$ (which is also its inverse):

$$Y = (W R^T) (R X)$$

We can pre-process the weights by computing $\hat{W} = W R^T$. At runtime, we transform the activations $\hat{X} = R X$. The result of the multiplication remains identical, but both $\hat{W}$ and $\hat{X}$ are now "smoothed" and ready for low-bit quantization.

Xijie Huang and Shih-Yang Liu demonstrate in RoLoRA that this rotation can even be integrated into fine-tuning, allowing LoRA adapters to work effectively in 4-bit weight-activation (W4A4) settings.

From Random to Learned Transformations

While random rotations like the Hadamard transform are computationally cheap, they are "data-blind." Recent work has moved toward optimizing these transformations to fit the specific statistical "landscape" of a model.

  • Kurtosis Minimization: Mohammad Sadegh Akhondzadeh introduces KurTail, which uses kurtosis—a measure of "tailedness"—to optimize rotations. By minimizing kurtosis, they explicitly suppress the outliers that cause quantization error.
  • Learned Rotations: Zechun Liu and the SpinQuant team show that learning the rotation matrices through optimization significantly outperforms fixed transforms, bridging the gap between low-precision and full-precision models.
  • Data-Aware Reshaping: Jiale Chen and Dan Alistarh propose WUSH, which uses weight-activation covariance to find near-optimal linear transforms, outperforming "blind" Hadamard rotations.

Hardware Acceleration and Diffusion

The overhead of performing these rotations at runtime is a concern, but researchers are finding ways to make it nearly free. Mudhakar Srivatsa and colleagues developed HadaCore, a GPU kernel that repurposes NVIDIA's Tensor Cores to accelerate the Hadamard transform, achieving up to 3.6x speedups.

This direction is also proving essential for Diffusion Transformers (DiTs). Unlike LLMs, DiT activations change significantly across different diffusion timesteps. Marco Federici and Markus Nagel introduced HadaNorm, which combines mean-centering with Hadamard transforms to "whiten" these shifting distributions. Similarly, RobuQ by Kaicheng Yang uses these transforms to push DiTs to extreme W1.58A2 (ternary weight, 2-bit activation) quantization while maintaining image quality.

Summary of Intuition

Incoherence processing works because it decouples the information in a tensor from its numerical representation. By rotating the data into a basis where no single dimension is "special," we prevent outliers from hijacking the quantization scale. This transformation turns the difficult problem of "spiky" quantization into the well-solved problem of Gaussian quantization, enabling the deployment of massive models on consumer-grade hardware.

Papers in this version

Showing 60 of 239 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 186 papers
  2. Version 1 · 2026-10-04 · 239 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_incoherence_processing_8e6a53a8,
  title = {incoherence processing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/incoherence-processing/rdr_8e6a53a838254fc4a08cec14c4d9c8f5},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_incoherence_processing_for_low_bit_quant_8e6a53a8_v1,
  title = {Incoherence Processing for Low-Bit Quantization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/incoherence-processing-for-low-bit-quantization/rdr_8e6a53a838254fc4a08cec14c4d9c8f5/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}