POT-PTQ: A Two-step Power-of-Two Post-training for LLMs

Xinyu Wang, Vahid Partovi Nia, Peng Lu, Jerry Huang, Xiao-Wen Chang, Boxing Chen, Yufei Cui

Abstract

Large Language Models (LLMs) have demonstrated re- markable performance across various natural language processing (NLP) tasks. However, their deployment is challenging due to the substantial computational resources required. Power-of-two (PoT) quantization is a general tool to counteract this difficulty. Albeit pre- vious works on PoT quantization can be efficiently dequantized on CPUs using fixed-point addition, it showed less effectiveness on GPUs. The reason is entanglement of the sign bit and sequential bit manipula- tions needed for dequantization. We propose a novel POT quantization framework for LLM weights that (i) outperforms state-of-the-art ac- curacy in extremely low-precision number formats, and (ii) enables faster inference through more efficient dequantization. To maintain the accuracy of the quantized model, we introduce a two-step post- training algorithm: (i) initialize the quantization scales with a robust starting point, and (ii) refine these scales using a minimal calibration set. The performance of our PoT post-training algorithm surpasses the current state-of-the-art in integer quantization, particularly at low precisions such as 2- and 3-bit formats. Our PoT quantization acceler- ates the dequantization step required for the floating point inference and leads to 3.67×speed up on a NVIDIA V100, and 1.63×on a NVIDIA RTX 4090, compared to uniform integer dequantization.
BibTeX
@inproceedings{wang2025pot,
  author = {Xinyu Wang and Vahid Partovi Nia and Peng Lu and Jerry Huang and Xiao-Wen Chang and Boxing Chen and Yufei Cui},
  title = {{POT-PTQ: A Two-step Power-of-Two Post-training for LLMs}},
  booktitle = {The European Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=mFB2M1XCKj},
}
Figure