TL;DR
This paper presents a comprehensive evaluation of the HiFloat family (HiF8 and HiF4)—low-bit floating-point formats specifically tailored for Ascend NPUs—in the context of Large Language Model (LLM) post-training quantization (PTQ). The authors demonstrate that while uniform integer formats (INT8) excel for narrow-range weights, HiFloat formats are uniquely suited for high-variance activations and KV-caches, with 4-bit HiF4 successfully preventing the catastrophic accuracy collapse typical of 4-bit integer methods.
What Problem Does This Paper Address?
As Large Language Models scale up, they place immense pressure on memory bandwidth and computational throughput. Quantization—reducing the bit-width of weights and activations—is a go-to strategy to alleviate these bottlenecks. While integer-based quantization (like INT8 and INT4) is mature, it struggles to handle the high dynamic volatility and heavy-tailed outlier distributions common in modern LLM activations.
To address this, the industry is pivoting toward low-bit floating-point formats (such as MXFP4 and NVFP4). However, their efficacy in post-training quantization (PTQ) pipelines on specialized hardware like Ascend NPUs has remained largely unexamined. This paper fills that gap by evaluating the HiFloat format family across the entire inference pipeline, including weights, activations, and key-value (KV) caches.
Methods
The authors provide formal quantization and dequantization logic for two primary formats:
- HiF8 (8-bit HiFloat): Unlike static floating-point formats, HiF8 uses a dynamic prefix code to adapt its bit-field allocation based on numerical distributions. It dynamically shifts the bit split between the exponent and mantissa depending on the magnitude of the data. To optimize it for static, bounded weight distributions, the authors introduce a per-channel scaling mechanism called HiF8 (scaled).
- HiF4 (4-bit HiFloat): Designed for ultra-low bit-widths, HiF4 utilizes a three-level hierarchical block-based scaling scheme. Each 64-element block shares an 8-bit unsigned scale factor, which is progressively refined at the 8-element sub-block and 4-element micro-block levels. Crucially, this mathematical formulation allows the hardware to reconstruct values using simple integer addition and bit shifts, bypassing expensive floating-point multiplications.
The authors also evaluate how these formats synergize with state-of-the-art outlier-mitigation PTQ frameworks like SmoothQuant and SVDQuant.
Datasets
The formats were evaluated using prominent open-source LLMs (Qwen3-8B and openPangu-7B) across several standard benchmarks:
- Language Modeling Perplexity: Wikitext-2 and C4.
- Zero-Shot Accuracy Suite: ARC Challenge, HellaSwag, MMLU, Math-500, and GSM8K.
- Long-Context Evaluation: LongBench (to stress-test KV cache quantization over extended sequences).
Key Insights and Ideas
- The Weight-Activation Bifurcation: In the 8-bit regime, there is a clear split in what works best. Denser, uniform integer formats (INT8) are superior for weights because weights concentrate heavily in a narrow, bounded range; using floating-point formats here results in "exponent waste." Conversely, floating-point formats are far better at capturing volatile activation distributions and extreme outliers.
- Hierarchical Scaling Rescues 4-bit Precision: Dropping to 4-bit uniform integer spacing (INT4) causes a total collapse in local feature resolution. HiF4’s multi-level hierarchical scaling acts as a critical safety net, isolating local variance and extreme outliers without needing a global FP32 scaling factor.
- Algorithmic Synergy: HiFloat formats are fully compatible with advanced PTQ frameworks. When paired with SmoothQuant or SVDQuant, the combined approach effectively mitigates outlier spikes, allowing low-bit formats to near-completely recover baseline performance.
Key Results
- W8A8 Quantization: Models showed high resilience. For Qwen3-8B, all evaluated formats maintained over 99% of the original BF16 baseline accuracy. When integrated with SmoothQuant, HiF8 effectively matched the baseline performance.
- W4A4 Quantization: While INT4 failed catastrophically, HiF4 maintained remarkable model integrity. On openPangu-7B, HiF4 limited accuracy degradation to just 3.0% under basic round-to-nearest (RTN) quantization, which narrowed to 2.7% when enhanced with SmoothQuant.
- KV Cache Performance: HiF4 achieved state-of-the-art performance in 4-bit KV cache settings. On Qwen3-8B, it restricted accuracy losses to 3.15% for Q16KV4 and 3.92% for QKV4, noticeably outperforming alternative formats like MXFP4 and NVFP4 on long-context tasks.
Limitations and Future Work
While the paper provides a rigorous mathematical and algorithmic evaluation of HiFloat formats—demonstrating excellent preservation of model accuracy (SQNR and perplexity)—it does not include physical hardware execution metrics, such as actual latency speedups or power efficiency measurements on physical Ascend NPUs. The authors imply that hardware deployment details and native kernel optimizations remain an open avenue for future engineering and evaluation.
Summary updated