Block Rotation is All You Need for MXFP4 Quantization

Yuantian Shao, Peisong Wang, Yuanteng Chen, Chang Xu, Zhihui Wei, Jian Cheng

TL;DR

While rotation-based techniques are the gold standard for 4-bit quantization in integer formats, they catastrophically fail on the emerging, hardware-backed MXFP4 floating-point format. This paper identifies a fundamental structural mismatch between global rotations and block-based scaling, and introduces Block-wise Rotation Quantization (BRQ)—a simple, elegant fix that restores model accuracy while cutting online deployment latency by 40%.

What Problem Does This Paper Address?

To run massive Large Language Models (LLMs) on commodity hardware, researchers use Post-Training Quantization (PTQ) to compress weights and activations down to 4 bits (W4A4). Recently, hardware giants like NVIDIA, AMD, and Intel have begun natively supporting MXFP4 (Microscaling FP4), an open-standard 4-bit floating-point format. MXFP4 groups numbers into blocks of 32 and assigns each block a shared Power-of-Two (PoT) scaling factor to manage extreme values (outliers).

State-of-the-art quantization algorithms typically rely on global rotation transforms (like Hadamard transforms) to smear out massive activation outliers evenly across all channels before quantization. However, when researchers tried combining these rotation methods with MXFP4, model performance collapsed. This paper investigates why this happens and provides a solution to make rotation techniques compatible with next-generation low-precision hardware.

Methods

The authors propose Block-wise Rotation Quantization (BRQ). Instead of applying a massive global rotation matrix that mixes data across all channels, BRQ breaks the activation channels into small, independent groups that perfectly align with MXFP4's hardware block size (e.g., 32 channels).

An independent orthogonal transformation is then performed strictly within each block. Mathematically, this replaces a dense global rotation matrix with a clean, block-diagonal matrix:

$$R_{\text{block}} = \text{diag}(R_1, R_2, \dots, R_B)$$

This localizes the outlier energy. Outliers are still smoothed out so they don't ruin their local block, but their energy never leaks into neighboring blocks. As a bonus, changing the operation from global to block-wise drops the online computational complexity from $O(N^2)$ to $O(N \times 32)$, making it highly hardware-friendly.

Visualizing how global rotation spreads outliers everywhere, whereas block-wise rotation keeps them contained locally.

Datasets

The method was benchmarked using:

  • WikiText2: Used to evaluate language modeling perplexity (where lower is better).
  • Five Zero-Shot Downstream Tasks: PIQA, WinoGrande, OpenBookQA, ARC-Easy, and ARC-Challenge to measure actual task accuracy.

These evaluations spanned multiple prominent open-source model families at various scales, including LLaMA-2 (7B/13B/70B), LLaMA-3 (8B), LLaMA-3.2 (1B/3B), Mistral-7B, and Qwen2.5 (1.5B/3B/7B).

Key Insights and Ideas

  • The "Regular Block" Sabotage: Global rotation successfully shrinks massive outliers, but because it preserves overall energy, it slightly inflates the values of the thousands of "regular" channels that originally had no outliers.
  • The PoT Penalty: MXFP4 uses coarse power-of-two steps at larger magnitudes. When global rotation amplifies originally small values, they hit these coarse steps, resulting in massive rounding errors. Since regular blocks vastly outnumber outlier blocks, these accumulated errors dominate and destroy model accuracy.
Figure 2: Effect of rotation and its variants across different quantization formats. Rot applies a ran- dom Hadamard transform with RTN; Rot+GPTQ combines the transform with GPTQ; and Opt. Rot+GPTQ employs an optimized rotation ma- trix with GPTQ.
  • Perfect Alignment Matters: The optimal rotation dimension size is not infinite; it is exactly equal to the hardware block size of the quantization format (32). Keeping rotations local completely mitigates over 80% of the scale inflation caused by global mixing.

Key Results

BRQ consistently rescues rotation-based PTQ under the MXFP4 format, frequently outperforming standard integer baselines (BINT4):

  • Perplexity Improvements: On the compact LLaMA-3.2 1B model, standard global rotation (QuaRot+) achieved a perplexity of 12.78. BRQ successfully reduced this to 11.95.
  • Downstream Accuracy: For Qwen2.5 1.5B, BRQ boosted average zero-shot accuracy from 53.50% (with global rotation) to 54.83%.
  • Inference Speedup: Because block-wise operations require far fewer computations than global ones, BRQ reduced the additional inference latency caused by rotation processing by 40% during the prefill stage on NVIDIA A800 evaluations.
Figure 1: Overall performance of quantization methods under MXFP4. The x-axis shows per- plexity, the y-axis shows average downstream ac- curacy, and methods nearer the top-left are closer to the FP16 baseline, indicating better perfor- mance.

Limitations and Future Work

While BRQ works exceptionally well out of the box using randomized block rotations, the authors noted that further end-to-end optimization of these rotation matrices yielded diminishing returns on certain architectures (like LLaMA-3 8B and Mistral 7B). This suggests current optimization frameworks have inherent limits when dealing with non-uniform scaling.

Future work will focus on designing better rotation matrix optimization schemes and exploring whether the online fast Hadamard transform can be entirely replaced by pre-optimized block rotations to further minimize runtime latency.

Summary updated

Abstract

Large language models (LLMs) have achieved remarkable success, but their rapidly growing scale imposes prohibitive costs in memory, computation, and energy. Post-training quantization (PTQ) is a promising solution for efficient deployment, yet achieving accurate W4A4 quantization remains an open challenge. While most existing methods are designed for INT4 formats, the emergence of MXFP4 -- a new FP4 format with various hardware support (NVIDIA, AMD, Intel)-- raises questions about the applicability of current techniques. In this work, we establish a comprehensive benchmark of PTQ methods under the MXFP4 format. Through systematic evaluation, we find that methods like GPTQ consistently deliver strong performance, whereas rotation-based approaches, which are almost used by all state-of-the-art approaches, suffer from severe incompatibility with MXFP4. We further provide the first in-depth analysis of this conflict, tracing its root to a fundamental mismatch between MXFP4's PoT (power-of-two) block scaling and the redistribution of outlier energy via global rotation. Building on this insight, we propose a simple yet effective block rotation strategy that adapts rotation-based methods to MXFP4, leading to substantial accuracy improvements across diverse LLMs. Our findings not only offer clear guidance for practitioners but also set a foundation for advancing PTQ research under emerging low-precision formats.
BibTeX
@inproceedings{shao2026block,
  author = {Yuantian Shao and Peisong Wang and Yuanteng Chen and Chang Xu and Zhihui Wei and Jian Cheng},
  title = {{Block Rotation is All You Need for MXFP4 Quantization}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=cyR2FtZzY1},
}
Figure