Calibration and Transformation-Free Weight-Only LLMs Quantization via Dynamic Grouping

Xiaolei Zheng, Zhen-Qun Yang, Haoran Xie, Shaorong Qin, A. Chen, Fangzhen Lin, Lin, FangzhenarXiv 2025 · DOI · arXiv

Abstract

Large Language Models (LLMs) deliver strong performance but are difficult to deploy under tight memory and compute constraints. Low-bit post-training quantization (PTQ) is a promising direction; however, it typically relies on calibration data, auxiliary transformations, and GPU tools. To address these limitations, we propose MSB (Multi Scale Binary), a calibration-free and transformation-free PTQ method that generalizes binary quantization to multi-bit settings. MSB optimizes a dynamic grouping criterion that minimizes within-group variance, yielding group-wise multiscale levels that can be applied consistently across granularities from per-tensor to block-wise configurations with 64 elements groups per row, without calibration or intermediate transforms. We implement the optimization in a CPU-based solver for the quantization step and evaluate using standard bfloat16 execution without low-bit packing. On Llama 3.2 3B, MSB achieves 8.43 perplexity on WikiText-2 under 4-bit weight-only block-wise quantization, compared to 7.81 in full precision and 12.23 with GPTQ in its default setup. Overall, MSB provides a new optimization perspective for low-bit PTQ while simplifying the pipeline by removing calibration and transformations.

Introduction

Large Language Models (LLMs) are massive, making deployment under tight memory and compute constraints incredibly expensive. Post-training quantization (PTQ)—specifically 4-bit weight-only quantization—is a standard solution to shrink the memory footprint of these models without the massive compute costs of retraining. However, current state-of-the-art PTQ pipelines (like GPTQ or AWQ) typically rely on calibration data to estimate sensitivity and complex mathematical transformations (like rotations or scaling) to smooth out outlier weights. Relying on calibration data can lead to overfitting, and transformations add complex design dependencies. MSB (Multi Scale Binary) addresses these issues by achieving highly accurate low-bit quantization directly from pre-trained weights, without relying on any data or transformations.

Methodology: Multi Scale Binary (MSB)

MSB is a calibration-free and transformation-free PTQ framework. Instead of applying a uniform grid to round weights, MSB generalizes 1-bit (binary) quantization principles to multi-bit settings. The core idea is to dynamically partition weights into unstructured groups where weights share a single scale parameter ($\alpha$) but retain individual binary signs ($\pm 1$).

The MSB quantization pipeline: (a) Original FP32 matrix, (b) Dynamic grouping to minimize variance, and (c) Resulting binary quantized weights.

To find the optimal groups, MSB minimizes the variance of absolute weight values within each group while penalizing the formation of excessively small groups. Since finding the perfect grouping is computationally expensive ($O(N^3)$), the authors developed practical heuristic solvers:

  • Windowed Greedy Merging (WGM): A fast approximation that restricts initial merging to local windows, used for 4-bit block-wise experiments.
  • Local Optimizing WGM (WGM-LO): Adds a stochastic local search to refine group boundaries, primarily used for per-tensor quantization.

Experimental Evaluation

The performance of MSB was evaluated across several model families, including LLaMA 3.2, Falcon 3, and Gemma 3, using standard NLP benchmarks for zero-shot common sense QA and perplexity (PPL).

Performance comparison of MSB (WGM/WGM-LO) against standard PTQ methods across various LLMs in 4-bit blockwise and 6-bit per-tensor settings.

Key Results

  • Competitive 4-bit Accuracy: On LLaMA 3.2 3B (4-bit block-wise), MSB achieves a WikiText-2 perplexity of 8.43, trailing the full-precision baseline (7.81) but significantly outperforming GPTQ (12.23) and bitsandbytes (8.28).
  • Robustness Across Architectures: MSB consistently ranked top-1 or top-2 across tested models. Notably, while GPTQ performance degraded on Llama 3.2 1B due to calibration sensitivity, MSB remained stable because it is data-free.
  • Granularity Agnostic: The same MSB objective works seamlessly for both 6-bit per-tensor and 4-bit block-wise granularities without requiring setting-specific adjustments.

Discussion and Conclusion

MSB demonstrates that calibration data and complex transformations are not strictly necessary to match state-of-the-art 4-bit performance. By focusing on a distribution-sensitive grouping of weights to minimize variance, MSB provides a robust, CPU-friendly quantization pipeline.

While current evaluations use simulated quantization (storing values in bfloat16), future work involves developing specialized low-bit CUDA/CPU kernels to realize actual hardware speedups. Additionally, the computational cost of the WGM solver remains higher than simple rounding, suggesting a need for further optimization in approximation algorithms.

Summary updated

Abstract

Large Language Models (LLMs) deliver strong performance but are difficult to deploy under tight memory and compute constraints. Low-bit post-training quantization (PTQ) is a promising direction; however, it typically relies on calibration data, auxiliary transformations, and GPU tools. To address these limitations, we propose MSB (Multi Scale Binary), a calibration-free and transformation-free PTQ method that generalizes binary quantization to multi-bit settings. MSB optimizes a dynamic grouping criterion that minimizes within group variance, yielding group-wise multiscale levels that can be applied consistently across granularities from per tensor to block-wise configurations with 64 elements groups per row, without calibration or intermediate transforms. We implement the optimization in a CPU based solver for the quantization step and evaluate using standard bfloat16 execution without low-bit packing. On Llama 3.2 3B, MSB achieves 8.43 perplexity on WikiText-2 under 4-bit weight only block-wise quantization, compared to 7.81 in full precision and 12.23 with GPTQ its default setup. Overall, MSB provides a new optimization perspective for low-bit PTQ while simplifying the pipeline by removing calibration and transformations.
BibTeX
@misc{zheng2025calibration,
  author = {Xiaolei Zheng and Zhen-Qun Yang and Haoran Xie and Shaorong Qin and A. Chen and Fangzhen Lin and Lin, Fangzhen},
  title = {{Calibration and Transformation-Free Weight-Only LLMs Quantization via Dynamic Grouping}},
  year = {2025},
  eprint = {2509.03054},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.03054},
  howpublished = {arXiv},
}
Figure