TL;DR
This paper introduces three novel calibration methods—HiGPTQ, HiAutoRound, and HiFlatQuant—specifically designed for the HiFloat4 (HiF4) 4-bit quantization format. By adapting popular post-training quantization (PTQ) techniques to HiF4’s unique hierarchical structure, the authors reduce the accuracy gap between 4-bit models and their 16-bit counterparts to less than 1%, outperforming existing industry standards like NVFP4.
What Problem Does This Paper Address?
As Large Language Models (LLMs) grow, the pressure to compress them for deployment increases. Moving from 8-bit to 4-bit quantization can halve memory usage, but it often comes at a steep cost to model accuracy.
Standard 4-bit integer formats (like INT4) often struggle to represent the vast dynamic range of LLM weights and activations. To solve this, new formats like HiFloat4 (HiF4) have emerged. HiF4 uses a "Block Floating-Point" (BFP) approach with a three-level scaling hierarchy, allowing it to capture both tiny and huge values more effectively than simple integers.
However, existing calibration tools—the "fine-tuning" step that minimizes quantization error—were built for simple integer formats. They don't understand HiF4's complex, shared scaling structure. This paper bridges that gap by redesigning calibration algorithms to be "HiF4-aware."
Methods
The authors propose three distinct calibration strategies, each adapting a well-known PTQ algorithm to the HiF4 format:
- HiGPTQ: Based on the GPTQ algorithm, which uses second-order information (the Hessian matrix) to compensate for quantization errors. The "Hi" version modifies the process to quantize weights in groups of 64, matching HiF4’s shared scaling metadata (see Figure 2a).
- HiAutoRound: This method introduces learnable parameters to adjust how weights are rounded. Specifically, it learns offsets for the micro-exponents ($K$) and mantissas ($M$) within the HiF4 block to minimize the difference between the original and quantized outputs (see Figure 2b).
- HiFlatQuant: This approach learns affine transformations to "flatten" the distribution of weights and activations, making them easier to quantize without losing information to outliers. Interestingly, because HiF4 has a naturally large dynamic range, the authors found they could omit the complex "learnable clipping" required for integer formats, simplifying the process (see Figure 2c).
Datasets
The researchers evaluated their methods across five major LLM families (Llama2-7B/13B, Llama3-8B, and Qwen2.5-7B/14B) using a suite of standard benchmarks:
- Common Sense & Reasoning: ARC-Challenge, ARC-Easy, BoolQ, HellaSwag, Piqa, and WinoGrande.
- Language Modeling: Lambada (OpenAI version).
- Calibration Data: A small subset (128 sentences) of the WikiText2 training set was used to calibrate the models.
Key Insights and Ideas
- Hierarchy Matters: HiF4’s three-level scaling (Level-1 FP8, Level-2/3 micro-exponents) is inherently superior to 4-bit formats like MXFP4 and NVFP4 because it provides a much broader dynamic range.
- Structural Alignment: Calibration isn't one-size-fits-all. For HiGPTQ to work, the quantization block size must be a multiple of the HiF4 group size (64). This ensures the shared scaling factors are derived from the same elements the algorithm is currently optimizing.
- Outlier Resilience: One of the most interesting findings is that HiF4 is so robust to outliers that the "learnable clipping" parameters used in FlatQuant for INT4 are unnecessary. This reduces calibration complexity while maintaining high accuracy.
- HiGPTQ is the All-Rounder: While all three methods improved performance, HiGPTQ generally provided the most consistent gains across different models and tasks (see Figure 3).
Key Results
The experimental results demonstrate a clear win for HiF4-based calibration:
- Closing the Gap: Across all models, the proposed methods reduced the accuracy gap between 4-bit HiF4 and 16-bit BF16 to less than 1%.
- Superiority over NVFP4: HiF4 calibrated with HiGPTQ outperformed the state-of-the-art NVFP4 format. For example, on Llama2-13B, HiGPTQ achieved an average accuracy of 73.86%, actually slightly surpassing the BF16 baseline of 73.71% in some instances.
- Consistency: The improvements were not limited to one model; they were observed across Llama and Qwen architectures, proving the methods are generalizable.
Limitations and Future Work
- Inference Overhead: The paper notes that HiFlatQuant's performance depends on "transformation folding." If the learned transformations cannot be efficiently folded into existing layers, it could introduce inference latency.
- Hardware Specifics: While the experiments were conducted on simulated 4-bit formats on Huawei Ascend NPUs, the paper does not provide extensive real-world wall-clock speedup data for diverse hardware targets.
- PTQ vs. QAT: The study focuses entirely on Post-Training Quantization. Exploring how these hierarchical formats behave in Quantization-Aware Training (QAT) remains an open area for research.
Summary updated