PoTPTQ: A Two-step Power-of-Two Post-training for LLMs

Xinyu Wang, Vahid Partovi Nia, Peng Lu, Jerry Huang, Xiao-Wen Chang, Boxing Chen, Yufei Cui

TL;DR

PoTPTQ introduces a highly efficient Power-of-Two (PoT) post-training quantization framework designed to compress Large Language Models (LLMs) down to extreme 2-bit and 3-bit precisions without tanking their accuracy. By combining a robust two-step scale optimization strategy with a GPU-optimized integer-only dequantization kernel, it outperforms state-of-the-art uniform quantization methods while delivering up to a $3.67\times$ inference speedup on modern hardware.

What Problem Does This Paper Address?

As Large Language Models scale up, their massive memory and computational footprints make deployment on standard hardware incredibly challenging. Quantization—converting full-precision weights into lower-bit formats like INT4 or INT2—is a go-to solution, but aggressive uniform quantization often leads to severe accuracy degradation.

Standard uniform quantization distributes its representation levels evenly. However, LLM weights typically follow a bell-shaped, exponentially decaying distribution where the vast majority of weights cluster tightly around zero.

Figure 1: Left: Density distribution of the first weight matrix in LLaMA-7B, exemplifying the bell-shaped or exponential decay commonly observed in LLM weight distributions. Middle: Quantization levels for power-of-two (PoT) quantization, showing finer resolution near zero, aligning well with the distribution’s high-density region. Right: Quantization levels for uniform quantization, which allocate levels evenly and poorly capture the dense region near zero.

Power-of-Two (PoT) quantization solves this by spacing its quantization levels logarithmically ($\pm 2^E$), naturally allocating finer resolution near zero. Despite this statistical advantage, prior attempts to apply PoT to LLMs failed because:

  1. The discrete nature of PoT creates a highly non-smooth loss landscape, making standard gradient-based optimization fail.
  2. Naive dequantization on GPUs suffers from bit-level dependencies and sign-bit entanglement, introducing significant latency overhead during inference.

Methods

The authors introduce PoTPTQ, a framework that tackles both the optimization and hardware efficiency bottlenecks of PoT quantization through a two-step post-training algorithm and a custom GPU kernel.

Figure 2: The two step algorithm. Step 1: The scale is adjusted by aligning the weight matrices (left). Step 2: Activations are aligned using minimal calibration data (rights).

1. Data-Agnostic Scale Initialization (Step 1)

Because small variations in scale lead to abrupt, non-smooth shifts in exponent values, gradient descent is highly unreliable at the start.

Non-smooth loss landscape of PoT quantization

To circumvent this, the authors implement a parallelized grid search over a discrete set of scaling multipliers directly on the weight matrices, establishing a robust base scale for each weight group without needing activation data.

2. Data-Dependent Fine-Tuning (Step 2)

Once initialized, the scales are refined using a tiny calibration dataset. To keep computational costs low, the framework freezes the quantized exponents and learns a low-dimensional residual parameter ($\Gamma$) to adjust the scaling factors. This step optimizes an output-level alignment loss using the Straight-Through Estimator (STE) to bypass non-differentiable rounding operations.

3. GPU-Optimized Dequantization Kernel

During inference, low-bit weights must be dequantized back to FP16 for matrix multiplication (GEMM). PoTPTQ exploits the fact that multiplying an FP16 value by a power-of-two weight mathematically simplifies to adding the weight's exponent to the activation's exponent bits. The authors designed a custom kernel using fast bit manipulation and fixed-point integer addition, completely avoiding costly floating-point operations during the unpacking phase.

Datasets

The framework was evaluated on the LLaMA-1 and LLaMA-2 model families (ranging from 7B to 30B parameters) using:

  • WikiText-2: Used both as the evaluation benchmark for language modeling perplexity and as the source for the calibration set (128 randomly sampled 2048-token sequences) during the data-dependent fine-tuning step.
  • Open LLM Leaderboard Harness: A downstream evaluation suite spanning six standard QA and reasoning tasks (ARC-Challenge, ARC-Easy, BoolQ, Hellaswag, PIQA, and Winogrande) to verify that functional capabilities are preserved.

Key Insights and Ideas

  • Exploiting Natural Distributions: Uniform quantization wastes representation levels on outliers. PoT's logarithmic spacing inherently matches the heavy-tailed, zero-centered nature of neural network weights.
  • Grid Search Beats Gradients for Discrete Spaces: When dealing with highly non-smooth error surfaces caused by discrete rounding, a parallelized grid search is a more reliable initialization tool than gradient descent.
  • Bit-Level Additivity: By treating dequantization as an exponent-addition problem rather than a floating-point multiplication problem, hardware latency can be drastically reduced directly within GPU warp cycles.

Key Results

  • Superior Accuracy at Low Bits: At extreme bit-widths (2.25 and 3.25 average bits), PoTPTQ consistently achieved lower perplexity on WikiText-2 than strong uniform PTQ baselines like GPTQ, AWQ, and OmniQuant. For instance, on LLaMA1-7B at 2.25 bits, uniform baselines suffered from severe perplexity degradation, while PoTPTQ maintained a stable perplexity of 9.79.
  • Downstream Performance: On the Open LLM Leaderboard harness, PoTPTQ matched or outperformed OmniQuant across most reasoning tasks, proving it preserves broader functional capabilities.
  • Substantial Speedups: The custom integer-only dequantization kernel achieved a $3.67\times$ speedup on an NVIDIA V100 and a $1.63\times$ speedup on an NVIDIA RTX 4090 compared to standard uniform FP16 dequantization.
  • Time Efficiency: The entire quantization process is highly practical, taking only about 43 minutes for a LLaMA-7B model on a single commodity GPU.

Limitations and Future Work

  • Weight-Only Focus: The current framework applies strictly to weight-only quantization. While this significantly reduces memory bandwidth bottlenecks, extending PoT to activation quantization is required for a fully integer-only GEMM pipeline.
  • Diminishing Returns at Higher Bit-Widths: The structural advantages of PoT's logarithmic spacing are most pronounced at ultra-low precisions (2-bit and 3-bit). At higher bit-widths (e.g., 4-bit or 8-bit), standard uniform quantization is already highly optimized and may perform similarly with less specialized kernel overhead.

Summary updated

Abstract

Large Language Models (LLMs) have demonstrated remarkable performance across various natural language processing (NLP) tasks. However, their deployment is challenging due to the substantial computational resources required. Power-of-two (PoT) quantization is a general tool to counteract this difficulty. Albeit previous works on PoT quantization can be efficiently dequantized on CPUs using fixed-point addition, it showed less effectiveness on GPUs. The reason is entanglement of the sign bit and sequential bit manipulations needed for dequantization. We propose a novel POT quantization framework for LLM weights that (i) outperforms state-of-the-art accuracy in extremely low-precision number formats, and (ii) enables faster inference through more efficient dequantization. To maintain the accuracy of the quantized model, we introduce a two-step post-training algorithm: (i) initialize the quantization scales with a robust starting point, and (ii) refine these scales using a minimal calibration set. The performance of our PoT post-training algorithm surpasses the current state-of-the-art in integer quantization, particularly at low precisions such as 2- and 3-bit formats. Our PoT quantization accelerates the dequantization step required for the floating point inference and leads to $3.67\times$ speed up on a NVIDIA V100, and $1.63\times$ on a NVIDIA RTX 4090, compared to uniform integer dequantization.
BibTeX
@inproceedings{wang2025potptq,
  author = {Xinyu Wang and Vahid Partovi Nia and Peng Lu and Jerry Huang and Xiao-Wen Chang and Boxing Chen and Yufei Cui},
  title = {{PoTPTQ: A Two-step Power-of-Two Post-training for LLMs}},
  booktitle = {The European Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=mFB2M1XCKj},
}
Figure