Microscaling Formats for Low Precision Deep Learning

This direction is split into other directions: block-wise quantization. Its last version stays readable and citable here.

Version 1 of 1131 papers165 conceptsfrozen 2026-10-04

As deep learning models scale toward trillions of parameters, the computational cost of training and inference in standard 32-bit or 16-bit floating-point formats has become a primary bottleneck. While aggressive quantization to 4-bit or even 2-bit representations offers massive speedups, it traditionally suffers from significant accuracy degradation. The emerging solution, centered around Microscaling (MX) and Block Floating Point (BFP) formats, involves a fundamental shift in how we represent numerical tensors: moving from global scaling factors to fine-grained, block-based scaling.

The Problem of Numerical Outliers

The primary challenge in low-precision quantization is the presence of "outliers"—individual values with much larger magnitudes than the rest of the distribution. In a standard tensor-wide or channel-wide quantization scheme, a single outlier forces the scaling factor to be very large to avoid clipping. This "squashes" the remaining values into a tiny fraction of the available representational range, effectively turning them into zeros and destroying the model's signal.

To solve this, Mario Drumond, Martin Jaggi, and their colleagues introduced Hybrid Block Floating Point (HBFP). The core intuition is to partition tensors into small, contiguous tiles or blocks (e.g., 16 or 32 elements) that each share a single exponent. By localizing the scaling factor, an outlier only affects the precision of its immediate neighbors in its specific block, preserving the dynamic range for the rest of the tensor.

Microscaling as an Industry Standard

What began as a research concept has recently coalesced into a unified industry standard. Ritchie Zhao and a collaborative team from Microsoft, NVIDIA, Intel, and Meta introduced Microscaling (MX) Data Formats. These formats formalize the use of small, fixed-size groups of elements sharing a high-precision scale factor.

The power of MX formats lies in their flexibility. They can wrap around various underlying "mantissa" types, including integers (MXINT) and floating-point (MXFP). Mengzhao Chen and Zhihang Yuan have shown that at high granularities, fine-grained integer formats like MXINT8 can actually outperform floating-point counterparts in both accuracy and hardware efficiency.

Enabling 4-bit Training and Inference

The most significant impact of microscaling is its ability to stabilize ultra-low precision training. NVIDIA researchers recently demonstrated the first successful pretraining of a 12-billion-parameter LLM using 4-bit (NVFP4). By using a hierarchical quantization strategy—where small blocks share a scale, and those scales are themselves scaled by a global factor—they achieved accuracy parity with 8-bit baselines.

Similarly, Brian Chmiel and Daniel Soudry showed that fully quantized FP4 training is possible for weights, activations, and gradients. The intuition here is that the "group size" (the number of elements sharing a scale) acts as a critical lever. As Jing Liu notes in their work on scaling laws for QAT, reducing the group size allows the quantization process to adapt to local variances, effectively narrowing the performance gap between low-bit and full-precision models.

Hardware Efficiency and Throughput

Microscaling is not just a mathematical trick; it is designed for hardware. Traditional per-element scaling requires a dedicated exponent for every single number, which is memory-intensive. Global scaling is efficient but inaccurate. Microscaling provides a "middle ground" that maps perfectly to modern hardware like NVIDIA's Blackwell architecture.

Jintao Zhang and the creators of SageAttention3 leveraged these FP4 Tensor Cores to achieve massive throughput gains. By assigning independent scale factors to small 1x16 groups, they localized the impact of outliers in the attention mechanism, allowing for high-speed inference without the typical accuracy loss of 4-bit formats.

Beyond Standard Layers

The principle of fine-grained scaling is being applied across the entire deep learning stack:

  • Optimizers: Bingrui Li and Jun Zhu utilized 4-bit optimizer states by using small block sizes to adapt to local outlier patterns in gradients.
  • Convolutions: Researchers at Arm, including Gerti Tuzi, have applied group-wise quantization to Winograd convolutions, enabling faster CPU execution for diffusion models.
  • Post-Training Quantization (PTQ): Sayeh Sharify and Zifei Xu demonstrated that adapting algorithms like GPTQ to MX formats allows for near-lossless 4-bit weight compression.

By moving the "intelligence" of the numerical representation from the individual number to the local group, microscaling formats allow researchers to slash bit-widths while maintaining the nuanced dynamic range required by the world's largest AI models.

Papers in this version

Showing 60 of 131 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_microscaling_formats_for_low_precision_d_c5676f28,
  title = {Microscaling Formats for Low Precision Deep Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/microscaling-formats-for-low-precision-deep-learning/rdr_c5676f2859e84515831ea81c81e49dd0},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_microscaling_formats_for_low_precision_d_c5676f28_v1,
  title = {Microscaling Formats for Low Precision Deep Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/microscaling-formats-for-low-precision-deep-learning/rdr_c5676f2859e84515831ea81c81e49dd0/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}