Abstract
This study examines 4-bit quantization methods like GPTQ in large language models (LLMs), highlighting GPTQ's overfitting and limited enhancement in Zero-Shot tasks. While prior works merely focusing on zero-shot measurement, we extend task scope to more generative categories such as code generation and abstractive summarization, in which we found that INT4 quantization can significantly underperform. However, simply shifting to higher precision formats like FP6 has been particularly challenging, thus overlooked, due to poor performance caused by the lack of sophisticated integration and system acceleration strategies on current AI hardware. Our results show that FP6, even with a coarse-grain quantization scheme, performs robustly across various algorithms and tasks, demonstrating its superiority in accuracy and versatility. Notably, with the FP6 quantization, StarCoder-15B model performs comparably to its FP16 counterpart in code generation, and for smaller models like the 406M it closely matches their baselines in summarization. Neither can be achieved by INT4. To better accommodate various AI hardware and achieve the best system performance, we propose a novel 4+2 design for FP6 to achieve similar latency to the state-of-the-art INT4 fine-grain quantization. With our design, FP6 can become a promising solution to the current 4-bit quantization methods used in LLMs.
The Accuracy Gap in 4-bit Quantization
As Large Language Models (LLMs) scale, quantization—reducing model weights from 16-bit to lower precisions—becomes essential for efficient deployment. While 4-bit integer quantization (INT4) is the current industry standard, it exhibits significant fragility. Popular methods like GPTQ often "overfit" to their calibration datasets. For instance, an INT4 model calibrated on the C4 dataset may show acceptable perplexity on C4 but experience a performance collapse when evaluated on other datasets like PTB.
Furthermore, while INT4 may appear sufficient for simple zero-shot reasoning, it often fails in complex generative tasks such as code generation and abstractive summarization. This degradation is particularly acute for models with 13 billion parameters or fewer, which are critical for edge and local applications.
The ZeroQuant(4+2) Methodology
To address these limitations, we propose FP6 as the optimal "sweet spot" between compression and model quality. To overcome the hardware challenges of non-standard 6-bit widths, we introduce two key innovations:
- The 4+2 Bit-Splitting Design: We split each 6-bit weight into a 4-bit component (sign and exponent) and a 2-bit component (mantissa). This allows GPUs to load and dequantize these segments in parallel, bypassing the memory access bottlenecks typically associated with 6-bit data.
- Bias Shift Optimization: Traditional dequantization requires expensive runtime math to adjust exponent biases. Our "Bias Shift" technique integrates this adjustment into the scaling factor during a static pre-processing step, reducing runtime dequantization to a simple bit-level padding operation.
Experimental Results
Our evaluation across diverse tasks—including zero-shot reasoning, code generation (Human-X), and summarization (BART)—demonstrates the clear superiority of FP6 over INT4 variants.
Generative Task Performance
In complex generative tasks, FP6 consistently outperforms INT4 and even matches the performance of the original FP16 models. For the StarCoder-15B model, FP6 achieved performance virtually identical to the 16-bit baseline in code generation. Similarly, for smaller models like BART-406M, FP6 closely matched the full-precision baseline in summarization, a level of accuracy that neither INT4 nor FP5 could reach.
System Efficiency
Beyond accuracy, the ZeroQuant(4+2) implementation is highly competitive in terms of speed. Our optimized FP6 kernel achieves up to a 2.37x speedup over standard cuBLAS (FP16). More importantly, the 4+2 design allows FP6 to achieve inference latency similar to state-of-the-art INT4 fine-grained quantization (FGQ) implementations, such as those found in NVIDIA’s TensorRT-LLM, effectively removing the "hardware tax" previously associated with 6-bit precision.
Conclusion
ZeroQuant(4+2) redefines the landscape of LLM quantization by proving that FP6 is a more robust and versatile format than the standard INT4. By solving the hardware acceleration hurdles through the 4+2 design and Bias Shift optimization, we provide a solution that offers FP16-level accuracy for complex generative tasks while maintaining the deployment efficiency required for modern AI applications.
Summary updated