TL;DR
While rotation-based techniques are the gold standard for 4-bit quantization in integer formats, they catastrophically fail on the emerging, hardware-backed MXFP4 floating-point format. This paper identifies a fundamental structural mismatch between global rotations and block-based scaling, and introduces Block-wise Rotation Quantization (BRQ)—a simple, elegant fix that restores model accuracy while cutting online deployment latency by 40%.
What Problem Does This Paper Address?
To run massive Large Language Models (LLMs) on commodity hardware, researchers use Post-Training Quantization (PTQ) to compress weights and activations down to 4 bits (W4A4). Recently, hardware giants like NVIDIA, AMD, and Intel have begun natively supporting MXFP4 (Microscaling FP4), an open-standard 4-bit floating-point format. MXFP4 groups numbers into blocks of 32 and assigns each block a shared Power-of-Two (PoT) scaling factor to manage extreme values (outliers).
State-of-the-art quantization algorithms typically rely on global rotation transforms (like Hadamard transforms) to smear out massive activation outliers evenly across all channels before quantization. However, when researchers tried combining these rotation methods with MXFP4, model performance collapsed. This paper investigates why this happens and provides a solution to make rotation techniques compatible with next-generation low-precision hardware.
Methods
The authors propose Block-wise Rotation Quantization (BRQ). Instead of applying a massive global rotation matrix that mixes data across all channels, BRQ breaks the activation channels into small, independent groups that perfectly align with MXFP4's hardware block size (e.g., 32 channels).
An independent orthogonal transformation is then performed strictly within each block. Mathematically, this replaces a dense global rotation matrix with a clean, block-diagonal matrix:
$$R_{\text{block}} = \text{diag}(R_1, R_2, \dots, R_B)$$
This localizes the outlier energy. Outliers are still smoothed out so they don't ruin their local block, but their energy never leaks into neighboring blocks. As a bonus, changing the operation from global to block-wise drops the online computational complexity from $O(N^2)$ to $O(N \times 32)$, making it highly hardware-friendly.
Datasets
The method was benchmarked using:
- WikiText2: Used to evaluate language modeling perplexity (where lower is better).
- Five Zero-Shot Downstream Tasks: PIQA, WinoGrande, OpenBookQA, ARC-Easy, and ARC-Challenge to measure actual task accuracy.
These evaluations spanned multiple prominent open-source model families at various scales, including LLaMA-2 (7B/13B/70B), LLaMA-3 (8B), LLaMA-3.2 (1B/3B), Mistral-7B, and Qwen2.5 (1.5B/3B/7B).
Key Insights and Ideas
- The "Regular Block" Sabotage: Global rotation successfully shrinks massive outliers, but because it preserves overall energy, it slightly inflates the values of the thousands of "regular" channels that originally had no outliers.
- The PoT Penalty: MXFP4 uses coarse power-of-two steps at larger magnitudes. When global rotation amplifies originally small values, they hit these coarse steps, resulting in massive rounding errors. Since regular blocks vastly outnumber outlier blocks, these accumulated errors dominate and destroy model accuracy.
- Perfect Alignment Matters: The optimal rotation dimension size is not infinite; it is exactly equal to the hardware block size of the quantization format (32). Keeping rotations local completely mitigates over 80% of the scale inflation caused by global mixing.
Key Results
BRQ consistently rescues rotation-based PTQ under the MXFP4 format, frequently outperforming standard integer baselines (BINT4):
- Perplexity Improvements: On the compact LLaMA-3.2 1B model, standard global rotation (QuaRot+) achieved a perplexity of 12.78. BRQ successfully reduced this to 11.95.
- Downstream Accuracy: For Qwen2.5 1.5B, BRQ boosted average zero-shot accuracy from 53.50% (with global rotation) to 54.83%.
- Inference Speedup: Because block-wise operations require far fewer computations than global ones, BRQ reduced the additional inference latency caused by rotation processing by 40% during the prefill stage on NVIDIA A800 evaluations.
Limitations and Future Work
While BRQ works exceptionally well out of the box using randomized block rotations, the authors noted that further end-to-end optimization of these rotation matrices yielded diminishing returns on certain architectures (like LLaMA-3 8B and Mistral 7B). This suggests current optimization frameworks have inherent limits when dealing with non-uniform scaling.
Future work will focus on designing better rotation matrix optimization schemes and exploring whether the online fast Hadamard transform can be entirely replaced by pre-optimized block rotations to further minimize runtime latency.
Summary updated