Localizing Dynamic Range with Block-Wise Quantization

48 concepts37 papers1 proposal

Quantization compresses neural networks by mapping high-precision floating-point numbers to low-bitwidth representations. In a traditional uniform quantization scheme, a single scaling factor is calculated for an entire tensor or an entire channel. This approach works well if the underlying data is uniformly distributed. However, modern deep neural networks—especially Large Language Models (LLMs) and diffusion models—frequently exhibit "outlier" activations and weights with disproportionately large magnitudes.

When a single extreme outlier dictates the scaling factor for an entire tensor, the quantization grid stretches to accommodate it. As a result, the step size between discrete states becomes larger than the subtle variations of the remaining, smaller values. These smaller values are effectively rounded to zero, destroying the signal-to-noise ratio and leading to severe accuracy degradation.

Isolating Outliers via Block-Wise Partitioning

To resolve this trade-off between capturing fine details and accommodating outliers, researchers have shifted toward block-wise quantization (also referred to as microscaling or group-wise quantization). The core intuition is simple: instead of calculating a single global scale factor, the tensor is partitioned into small, contiguous, fixed-size blocks (such as 16 or 32 elements). Each block is then assigned its own independent scaling factor or exponent.

By confining the scale factor to a localized group of elements, the numerical impact of an outlier is strictly contained. As noted by Daniel Lo in Pushing the Limits of Narrow Precision Inferencing at Cloud Scale with Microsoft Floating Point, this mechanism ensures that high-dynamic-range values only affect the scaling of their immediate neighbors, preserving precision across the rest of the tensor. Early foundations of this idea, such as the Hybrid Block Floating Point (HBFP) framework introduced by Mario Drumond et al. in Training DNNs with Hybrid Block Floating Point, demonstrated that tiling tensors into smaller blocks with independent exponents prevents individual outliers from reducing the precision of an entire data structure, enabling stable training even with narrow bit-widths.

The Crossover Point: INT vs. FP at Fine Granularities

Historically, floating-point (FP) formats were preferred over integer (INT) formats for low-precision computing because their logarithmic spacing natively handles wide dynamic ranges. However, block-wise quantization fundamentally changes this dynamic.

When tensors are divided into small enough blocks, the local dynamic range within each block drops drastically. In these narrow windows, the distribution of values becomes highly uniform. Mengzhao Chen et al. in INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formats and INT vs. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formats demonstrated a critical "crossover point": at high granularities (such as block sizes of 16 or 32), uniform integer formats actually surpass floating-point formats in both representation accuracy and hardware efficiency. Because the local range is tightly bounded, uniform integer grids can map the local distribution more effectively than logarithmic scales, while avoiding the complex hardware logic required to execute floating-point arithmetic.

Managing the Metadata Overhead

The primary challenge of block-wise quantization is the overhead of storing and processing the extra scaling factors. If every group of 16 values requires a full 32-bit floating-point scale, the memory savings of low-bit quantization are quickly erased.

To make this approach practical for hardware accelerators, modern frameworks use highly compressed metadata strategies. For instance, Sifan Zhou in GSQ-Tuning: Group-Shared Exponents Integer in Fully Quantized Training for LLMs On-Device Fine-tuning introduces a group-shared exponent format that leverages the spatial locality of weight magnitudes. By sharing a single exponent across a block of adjacent values, the method minimizes the bit-budget required for exponents, preserving more bits for the mantissa and maintaining numerical stability during training. Similarly, microscaling (MX) formats employ lightweight 8-bit scale factors shared across small groups (e.g., 32 elements), balancing memory efficiency with the numerical stability required for hardware-accelerated tensor operations, as discussed by Wenyuan Liu et al. in MicroMix: Efficient Mixed-Precision Quantization with Microscaling Formats for Large Language Models.

Pushing the Limits of Ultra-Low Precision

As hardware architectures evolve to support ultra-low precision formats like FP4, block-wise scaling has become essential for preventing catastrophic accuracy loss. Carlo Luschi et al. in Elucidating the Design Space of FP4 training show that microscaling quantization provides the fine-grained control necessary to stabilize 4-bit training. This approach has been successfully mapped to next-generation hardware; for example, Jintao Zhang et al. in SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Training leverage block-wise microscaling to execute FP4 attention on modern Tensor Cores, localizing the impact of outliers and maintaining model accuracy despite the extremely limited representational capacity of 4-bit data types.

Beyond weights and activations, block-wise quantization has been applied across the entire deep learning stack:

Key Foundational Work

Training DNNs with Hybrid Block Floating Point (2018) — contributed 2 concept elements to this direction.
Communication-Efficient Distributed Blockwise Momentum SGD with Error-Feedback (2019) — contributed 1 concept elements to this direction.
SYQ: Learning Symmetric Quantization for Efficient Deep Neural Networks (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_block_wise_quantization_1cba1242,
  title = {block-wise quantization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/block-wise-quantization/rdr_1cba1242d6244d7cadff7e0ead5a7fe8},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_block_wise_quantization_1cba1242_v1,
  title = {block-wise quantization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/block-wise-quantization/rdr_1cba1242d6244d7cadff7e0ead5a7fe8/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake