Mixed Precision Quantization via Sensitivity Aware Allocation

768 concepts502 papers9 proposals

Quantizing a deep learning model—reducing the precision of its weights, activations, or key-value (KV) caches from 16-bit floating-point numbers to low-bit integers like INT4 or INT2—is one of the most effective ways to deploy massive models on resource-constrained hardware. However, a uniform "one-size-fits-all" quantization strategy often hits a physical wall. Forcing every parameter in a network down to an ultra-low bit-width frequently triggers representational collapse and severe accuracy degradation.

Mixed-precision quantization works because it rejects the assumption that all parameters contribute equally to a model's output. In any neural network, certain layers, channels, or tokens are highly sensitive to noise, while others are remarkably robust. By identifying these sensitive components and protecting them with higher precision while aggressively compressing the resilient remainder, mixed-precision quantization achieves extreme compression ratios with near-lossless performance.

The Intuition of Non-Uniform Sensitivity

Why do different parts of a network tolerate precision loss differently? The answer lies in the structural and informational density of the model's architecture. Early foundations laid by Huizi Mao et al. in Deep Compression: Compressing Deep Neural Networks with Pruning demonstrated that convolutional layers and fully connected layers exhibit fundamentally different tolerances to weight quantization. Similarly, Haotong Qin argues in BiPointNet: Binary Neural Network for Point Clouds that uniform binarization fails because it ignores the varying information density required across different layers.

In modern architectures like Transformers, this non-uniformity is even more pronounced. For instance, the down-projection layers in multi-layer perceptron (MLP) blocks or the initial boundary layers often act as informational bottlenecks. As shown by Saleh Ashkboos et al. in QUIK: Towards End-to-end 4-Bit Inference on Generative Large Language Models, keeping a tiny fraction of these highly sensitive outlier features in 8-bit precision while quantizing the rest of the network to 4-bit allows the model to maintain its reasoning capabilities while unlocking significant hardware speedups.

Measuring Sensitivity

To allocate bits effectively, a system must mathematically quantify how much a specific parameter group "cares" about precision loss. Researchers use several distinct metrics to diagnose this sensitivity:

Loss Curvature (The Hessian)

The Hessian matrix describes the local curvature of the loss function. If a layer has a high Hessian trace, it means even a minor perturbation in its weights will cause a massive spike in the overall loss. Ivan Lazarevich et al. leveraged this property in the Neural Network Compression Framework for fast model inference to automate mixed-precision configurations. By calculating the Hessian, the framework isolates layers with high curvature and keeps them at higher bit-widths.

This principle is widely applied across modalities. For speaker verification models, Bei Liu et al. utilized Hessian-based sensitivity analysis in Adaptive Neural Network Quantization For Lightweight Speaker Verification to protect critical layers. For large language models, Ziyi Guan et al. developed APTQ: Attention-aware Post-Training Mixed-Precision Quantization for Large Language Models, which uses the Hessian trace to capture the non-linear impacts of attention mechanisms when deciding layer-wise precision.

Taylor Expansions and Gradients

When computing the full Hessian is too computationally expensive, first-order approximations offer a lightweight alternative. Sijie Zhao et al. demonstrated in Distribution-Aware Adaptive Multi-Bit Quantization that a first-order Taylor expansion can effectively measure how individual channels react to precision reduction, allowing for rapid, automated bit allocation.

Output Divergence and Feature Diversity

Rather than measuring the impact on the final loss, some methods look at how much the internal representations change. Tianzi Wang et al. utilized Kullback–Leibler (KL) divergence in Lossless 4-bit Quantization of Architecture Compressed Conformer ASR Systems on the 300-hr Switchboard Corpus to track shifts in output distributions, ensuring that layers causing large distributional shifts receive more bits. In vision models, Yunhe Wang et al. introduced a method in Post-Training Quantization for Vision Transformer that computes the nuclear norm of feature maps to estimate feature diversity; layers exhibiting high diversity are prioritized for higher precision.

Solving the Allocation Problem

Once sensitivity is mapped across the network, the challenge shifts to a resource allocation problem: How do we distribute a finite budget of bits to minimize overall error?

This is frequently framed as a constrained optimization problem and solved using two primary paradigms:

  1. Integer Linear Programming (ILP): Systems formulate a discrete optimization problem to minimize gradient variance or quantization error under a strict memory budget. For example, Lianmin Zheng et al. developed GACT: Activation Compressed Training for Generic Network Architectures, which dynamically solves an integer programming problem at runtime to compress training activations. Similarly, Zikai Zhou et al. utilized ILP in LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits to optimize channel-wise precision during parameter-efficient fine-tuning.
  2. Pareto Frontier Optimization: By mapping the accuracy-to-memory trade-offs of various layer configurations, algorithms can traverse a Pareto frontier to select the most efficient model state. Yaohui Cai et al. used this approach in ZeroQ: A Novel Zero Shot Quantization Framework to determine optimal configurations without needing access to the original training data.

This optimization mindset allows models to break free from rigid integer boundaries (like choosing strictly between 2, 3, or 4 bits). As shown by Runsheng Bai et al. in SKIM: Any-bit Quantization Pushing The Limits of Post-Training Quantization, distributing bits fluidly based on channel sensitivity allows a model to achieve fractional average bit-widths (e.g., 3.5 bits) globally.

Granularity of Allocation

Mixed precision operates at multiple granularities depending on the architecture and the operational bottleneck being targeted:

Layer-Wise and Component-Selective

The coarsest approach varies precision layer by layer. In speech recognition, Shaojin Ding et al. demonstrated in 4-bit Conformer with Native Quantization Aware Training for Speech Recognition that keeping sensitive initial layers or attention blocks at 8-bit while pushing robust intermediate layers to 4-bit yields massive memory savings without performance drops.

Token and Cache Quantization

In long-context LLM inference, the Key-Value (KV) cache becomes the primary memory bottleneck. Rather than quantizing the entire cache uniformly, Yefei He et al. developed ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification, which identifies "salient tokens" critical to context retention and allocates them higher bit-widths.

Taking this further, Wei Tao developed MoQAE: Mixed-Precision Quantization for Long-Context LLM Inference via Mixture of Quantization-Aware Experts, which treats different quantization bit-widths (such as FP16, INT4, and INT2) as distinct "experts" within a Mixture-of-Experts framework. A lightweight router dynamically assigns token chunks to the appropriate precision expert based on their contextual importance.

Expert-Wise Quantization in Sparse Models

For Mixture-of-Experts (MoE) architectures, sensitivity aligns with how frequently an expert is routed to. Wei Huang et al. showed in Mixture Compressor for Mixture-of-Experts LLMs Gains More that evaluating an expert's importance using a combination of routing scores, activation frequency, and reconstruction error allows for highly effective heterogeneous bit allocation. This ensures that heavily utilized generalist experts remain pristine, while specialized, rarely triggered experts are compressed aggressively.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_mixed_precision_quantization_539353bb,
  title = {mixed-precision quantization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/mixed-precision-quantization/rdr_539353bbabf1425b9c5ed42d5ade6338},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_mixed_precision_quantization_539353bb_v1,
  title = {mixed-precision quantization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/mixed-precision-quantization/rdr_539353bbabf1425b9c5ed42d5ade6338/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake