Asymmetric Quantization for Skewed Activation Distributions

70 concepts34 papers1 proposal

When compressing deep neural networks via quantization, the goal is to map high-precision floating-point numbers (like FP16 or FP32) to a limited set of discrete integer bins (like INT4 or INT8). Standard quantization approaches typically rely on a symmetric grid centered exactly at zero. This means the quantization boundaries are forced to be equal and opposite: $[-X_{max}, X_{max}]$.

While a symmetric grid works remarkably well for neural network weights—which naturally tend to cluster symmetrically around zero—it fails catastrophically when applied to activations.

Modern neural network architectures rely heavily on non-linear activation functions that break symmetry. When you force a symmetric quantization grid onto highly asymmetric data, you encounter a fundamental mathematical inefficiency: you waste a massive portion of your limited bit-depth on empty numerical space, drastically increasing the rounding error for the values that actually matter.


The Intuition of the Wasteful Grid

To understand why this happens, consider a network using a Rectified Linear Unit (ReLU) activation function, which outputs strictly non-negative values $[0, X_{max}]$. If we apply a standard 4-bit signed symmetric quantizer to these activations, our available 16 integer bins are distributed uniformly across the range $[-X_{max}, X_{max}]$.

Because ReLU never produces negative values, the entire negative half of our quantization grid (8 out of the 16 bins) sits completely empty. As noted by Jie Hu et al. in their work on quantized convolutional networks, switching to an unsigned quantization scheme for non-negative activations effectively doubles the available representation range, allowing the grid to utilize its full bit-depth for positive values.

When we force a symmetric constraint on asymmetric distributions, the step size between our quantization levels ($\Delta$) must stretch to cover the empty negative space. A larger $\Delta$ directly translates to higher rounding errors. In ultra-low bit-widths (such as 2-bit, 3-bit, or 4-bit precision), this precision loss makes the difference between a highly accurate model and a completely broken one.


The Anatomy of Activation Asymmetry

Activation distributions vary wildly depending on the architecture and the task. Researchers have identified several distinct patterns of asymmetry that break traditional quantization assumptions:

  • Small Negative Tails: Activation functions like Swish, SiLU, and GELU do not clip strictly at zero; instead, they produce a small negative bump before flattening out. Yash Sanjay Bhalgat et al. (LSQ+) and Mingze Wang et al. (Q-YOLO) demonstrate that symmetric ranges either completely discard these crucial negative tails or waste significant bit-depth trying to balance them against a massive positive range.
  • Power-Law and Heavy-Tailed Distributions: In Vision Transformers (ViTs) and Large Language Models (LLMs), activations following Softmax and GELU layers exhibit extreme power-law characteristics. As observed by Zhuguanyu Wu et al. (AdaLog) and Zechun Liu et al. (SpinQuant), the vast majority of values cluster tightly near zero, while a few massive outlier dimensions stretch the range out.
  • Bimodal Bottlenecks: In specialized architectures like the Segment Anything Model (SAM), activations can cluster around two entirely separate, distant peaks. Chengtao Lv et al. (PTQ4SAM) show that uniform symmetric quantizers struggle here because they must span the entire gap between the peaks, leaving the sparsely populated middle region with high quantization error.
  • Normalization-Free Asymmetry: In low-level vision tasks like image super-resolution, networks often omit Batch Normalization layers. Mingbao Lin et al. (Dynamic Dual Trainable Bounds) and Zhijun Tu et al. show that without normalization to center the data, activations drift into highly skewed, non-centered distributions.

Shifting and Scaling the Grid

To solve this, the core idea of asymmetric quantization is to decouple the lower and upper bounds of the quantization grid, allowing it to dynamically shift and scale to match the actual data density.

1. Learnable Offsets and Shifts

Instead of assuming the grid is anchored at zero, we introduce a zero-point or offset parameter ($o$). The mapping function transforms from a symmetric formulation to an affine one:

$$x_{quant} = \text{clip}\left(\left\lfloor \frac{x}{s} \right\rfloor + o, \text{min\_bit}, \text{max\_bit}\right)$$

By making the offset $o$ and scale $s$ learnable parameters during Quantization-Aware Training (QAT), the grid can slide over to focus entirely on the active data region. Yash Sanjay Bhalgat et al. demonstrate this with LSQ+, where the grid shifts to preserve the small negative tails of Swish functions. Similarly, Eunhyeok Park et al. introduce trainable scale and shift parameters via their Differentiable Quantization (DuQ) framework to handle asymmetric mobile architectures.

2. Independent Upper and Lower Bounds

In post-training quantization (PTQ) settings where retraining isn't an option, researchers instead optimize the upper and lower clipping bounds independently. Mingbao Lin et al. utilize independent, trainable upper and lower bounds to prevent representational capacity loss in super-resolution networks.

When initializing these bounds for asymmetric distributions, Kai Liu et al. (2DQuant) show that it is highly effective to fix the lower bound tightly around the minimum value (e.g., near zero for GELU/Softmax) while performing a mean squared error (MSE) search exclusively on the upper bound to determine where to clip long-tail outliers.


Advanced Multi-Range and Microscaling Strategies

For extreme activation distributions—such as those found in LLMs and Vision Transformers—a single linear shift is often still insufficient. The field has evolved several advanced variants to handle these edge cases:

Twin Uniform Quantization

To handle the conflict between high-density regions near zero and sparse, high-magnitude attention scores, Zhihang Yuan et al. (PTQ4ViT) proposed using two overlapping quantization ranges with distinct scaling factors. One range is dedicated to resolving the high-density values near zero with high precision, while the second, wider range captures the critical outliers.

Symmetric Grid:  [-X_max ----------------------- 0 ----------------------- X_max]
                 [  Wasted Negative Space       |  Coarse Positive Resolution  ]

Twin Uniform:                                   [0 ---- Range 1 (Dense) ----]
                                                [0 ------------------ Range 2 (Outliers) ------------------]

Asymmetric Microscaling

Rather than computing a single scale factor for an entire tensor, modern formats operate on small blocks of numbers. Janghwan Lee et al. (AMXFP4) introduce separate scale factors for positive and negative values within a single microscaling block. This allows 4-bit floating-point formats to represent skewed activation distributions accurately without requiring expensive data rotations or calibration.

Unsigned Floating-Point Recovery

In diffusion models, where specific timesteps cause activations to become almost exclusively positive, Maosen Zhao et al. show that converting standard signed floating-point formats into unsigned formats with a zero-point offset recovers the representational capacity of the sign bit. This effectively reclaims an entire bit of precision that would otherwise be wasted.


The Hardware-Accuracy Trade-off

If asymmetric quantization is mathematically superior for activations, why isn't it used everywhere? The answer lies in hardware efficiency.

Symmetric quantization is highly optimized for hardware deployment. When both weights ($W$) and activations ($A$) are symmetric around zero, the integer matrix multiplication $W \times A$ can be executed directly on standard multiply-accumulate (MAC) engines without any extra overhead.

However, when activations are asymmetric, the introduction of the zero-point offset ($o_A$) introduces an extra term into the multiplication algebra:

$$W \times (A + o_A) = W \times A + o_A \sum W$$

This requires the hardware to compute a running sum of the weights ($\sum W$) and multiply it by the activation offset during inference, introducing a computational pipeline bottleneck.

To balance this trade-off, the consensus strategy across modern architectures is a hybrid approach: symmetric weights and asymmetric activations. As demonstrated by Yash Sanjay Bhalgat et al. and Lishun Wang et al., keeping weights symmetric preserves hardware efficiency, while allowing activations to be asymmetric captures the skewed nature of modern non-linearities, providing the optimal trade-off between hardware speed and model accuracy.

Key Foundational Work

Bayesian Network Classifiers with Reduced Precision Parameters (2012) — contributed 1 concept elements to this direction.
Generalized Batch Normalization: Towards Accelerating Deep Neural Networks (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_asymmetric_activation_quantization_674f1f67,
  title = {asymmetric activation quantization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/asymmetric-activation-quantization/rdr_674f1f67bd3e47e68bb64de98fe3addb},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_asymmetric_activation_quantization_674f1f67_v1,
  title = {asymmetric activation quantization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/asymmetric-activation-quantization/rdr_674f1f67bd3e47e68bb64de98fe3addb/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake