Calibration of Large Model Quantization based on HiFloat4 Format
Lists
Only you see this
Abstract
Quantization of Large Language Models (LLMs) has become increasingly important for reducing the computational costs and resource requirements for practical deployment. Growing interest in this area has motivated the community to move beyond 8-bit quantization towards 4-bit precision, leading to the development of customized 4-bit formats such as MXFP4 and NVFP4. Recently, HiFloat4 (HiF4) has emerged as a competitive alternative which adopts a Block Floating-Point (BFP) format with a three-level scaling hierarchy, thereby achieving a significantly larger dynamic range. While quantization can substantially reduce memory usage and accelerate inference, it frequently introduces accuracy degradation. Calibration is therefore essential for narrowing the performance gap between low-precision quantized models and their full-precision counterparts. Nevertheless, existing calibration methods are mainly designed and validated using integer-precision formats, such as INT4, lacking support for HiF4 due to its hierarchical BFP structure. In this paper, we investigate three calibration approaches, namely HiGPTQ, HiAutoRound, and HiFlatQuant, to further enhance model quantization based on the HiF4 format. Extensive experiment s demonstrate that all three proposed approaches improve inference accuracy compared with direct HiF4 conversion, reducing the gap between HiF4 and BF16 to less than 1% and outperforming the state-of-the-art NVFP4 format across various models and downstream tasks.
BibTeX
@misc{huang2026calibration,
author = {Jing Huang and Yu Cheng and Ziwei Yu and Lan Zhao and Sitan Li and Guangxiu Yao and Xin Wang and Yuanyong Luo and Hengyi Liang and Ting Li and Bo Liu and Hu Liu and Michael Bi Mi and Heng Liao},
title = {{Calibration of Large Model Quantization based on HiFloat4 Format}},
year = {2026},
eprint = {2610.41112},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.41112},
}