Robust Post Training Quantization for Distribution Shift

This direction is split into other directions: calibration data sensitivity. Its last version stays readable and citable here.

Version 1 of 1135 papers172 conceptsfrozen 2026-10-04

Post-training quantization (PTQ) is the standard recipe for squeezing massive models into smaller hardware. By converting 16-bit weights and activations into 4-bit or even 2-bit integers, researchers can slash memory usage and accelerate inference. However, most PTQ methods rely on a "calibration" step: they pass a small set of representative data through the model to calculate optimal scaling factors and identify outliers.

The hidden danger in this approach is calibration sensitivity. If the calibration data (e.g., general Wikipedia text) doesn't perfectly match the deployment environment (e.g., specialized medical records or Python code), the model's accuracy can collapse. This "distribution shift" occurs because the quantization parameters become overfitted to the statistical quirks of the calibration set.

The Calibration Trap

In traditional PTQ, researchers like Mohammad Mozaffari and Amir Yazdanbakhsh note in SLiM that one-shot compression depends heavily on input statistics. If these statistics are biased, the resulting model loses its general-purpose reasoning. Sean I. Young argues in Radio that treating compression as a formal rate-distortion problem can mitigate heuristics, but the risk of non-representative calibration data remains a fundamental bottleneck.

This sensitivity is particularly acute in Diffusion Models. As Tianao Zhang and Yong Guo demonstrate in Quant-dLLM, using clean text to calibrate a model that operates on noisy, masked signals during inference creates a massive performance gap. The model essentially learns to "see" in one light but is forced to operate in another.

Forcing Data into Better Shapes

One way to break the dependency on calibration data is to make the data itself more predictable. Instead of trying to find the perfect quantization grid for a messy, "spiky" distribution, researchers use mathematical transforms to force activations into a standard shape.

Amir Zandieh and Insu Han propose QJL, which uses random Johnson-Lindenstrauss projections to achieve "data-obliviousness." By projecting high-dimensional Key-Value (KV) caches into a lower-dimensional space using random Gaussian matrices, the method becomes immune to the specific distribution of the input text.

Similarly, Donghyun Son and Sungjoo Yoo introduced NSNQuant, which uses a "Normalize-Shift-Normalize" transformation combined with Hadamard rotations. This process mathematically "squeezes" outliers and centers the distribution, allowing for ultra-low 1-bit or 2-bit quantization that doesn't care whether the input is a poem or a piece of C++ code.

Data-Free and Zero-Shot Alternatives

If calibration data is the problem, some researchers suggest removing it entirely. Hanlin Tang and Yifu Sun developed EasyQuant, a data-free algorithm that identifies weight outliers and optimizes quantization ranges using only the weights themselves. This avoids the risk of overfitting to a specific dataset and preserves the model's original general-purpose reasoning.

In a similar vein, AdpQ by Alireza Ghaffari uses statistical shrinkage (Adaptive LASSO) to determine quantization thresholds. Because it doesn't "look" at calibration samples to set these thresholds, it remains robust across diverse tasks, from creative writing to technical programming.

Adapting at Runtime

The most flexible approach to distribution shift is to stop treating quantization as a static, one-time process. Instead, the model can adapt to the data it sees during actual use.

Jiaqi Zhao and Miao Zhang pioneered this with LRQuant, which introduces Test-Time Adaptation (TTA). Rather than relying solely on offline calibration, the model updates its smoothing parameters as it processes new, unseen data. This allows the quantized model to "settle in" to a new domain—like a specialized medical benchmark—even if it was never part of the original training or calibration set.

For hardware-constrained environments, A probabilistic framework for dynamic quantization suggests using lightweight surrogate models to predict the dynamic range of activations before they are computed. This provides the memory benefits of static quantization with the robustness of dynamic, input-adaptive methods, ensuring that even if the data distribution shifts wildly, the quantization grid shifts with it.

Papers in this version

Showing 60 of 135 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_robust_post_training_quantization_for_di_e4b96c45,
  title = {Robust Post Training Quantization for Distribution Shift},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/robust-post-training-quantization-for-distribution-shift/rdr_e4b96c4515624040a01332b4a1f203bd},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_robust_post_training_quantization_for_di_e4b96c45_v1,
  title = {Robust Post Training Quantization for Distribution Shift},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/robust-post-training-quantization-for-distribution-shift/rdr_e4b96c4515624040a01332b4a1f203bd/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}