Differentiable Search for Mixed Precision Quantization

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2292 papers386 conceptsfrozen 2026-10-04

Quantization—the process of mapping high-precision floating-point weights to low-bit integers—is the primary tool for deploying large models on edge devices. Traditionally, this was a manual process: an engineer might decide to use 8 bits for all weights or use heuristics to clip outliers.

However, a growing body of research suggests that quantization should not be a post-processing step, but a learnable part of the model's architecture. By relaxing discrete quantization choices into continuous, differentiable variables, researchers like Hsin-Pai Cheng and BoRui Wu have shown that models can "learn" their own optimal precision levels and scaling factors through standard gradient descent.

Relaxing the Discrete Bit-Width

The hardest part of optimizing quantization is that bit-width (e.g., choosing between 2, 4, or 8 bits) is a discrete choice. You cannot take the derivative of "4 bits" with respect to a loss function.

To solve this, researchers use a "super-net" approach. In Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search, the authors represent a single layer as a parallel set of possible bit-widths. During training, the model computes a weighted sum of these different precisions. By using the Gumbel-Softmax trick—a common tool in HMQ: Hardware Friendly Mixed Precision Quantization Block for CNNs—the model can gradually "harden" its choice, eventually converging on the single most efficient bit-width for that specific layer.

Other methods, such as FracBits and BitPruning, treat bit-width as a continuous value by interpolating between adjacent integer precisions. If the "fractional bit-width" is 3.5, the layer’s output is a 50/50 blend of 3-bit and 4-bit quantization. This allows Milos Nikolic and others to use standard backpropagation to find the exact point where a layer provides the most "accuracy per bit."

Learning the Quantization Grid

Even if the bit-width is fixed, the range of the quantization grid matters. If you quantize a weight distribution from -10 to 10 into 4 bits, but 99% of your weights are between -1 and 1, you waste almost all your precision on empty space.

Instead of using fixed clipping ranges, Adaptive Low-Precision Training (ALPT) and GTC treat the "step size" (the distance between discrete levels) as a trainable parameter. As Shiwei Li demonstrates, the model can then automatically squeeze or stretch its quantization grid to fit the actual distribution of weights and activations.

This is particularly powerful for handling "outliers"—extreme values that often break simple quantization schemes. By making the clipping threshold a learnable parameter, as seen in BASQ, the model learns to ignore a few extreme outliers to achieve better resolution for the majority of its weights.

Scaling to Large Language Models

As models grow to billions of parameters, the cost of "re-training" for quantization becomes prohibitive. Recent work has shifted toward "calibration-driven" optimization. In TEQ: Trainable Equivalent Transformation for Quantization of LLMs, the authors show that training less than 0.1% of a model's parameters—specifically the scaling factors—is enough to recover the accuracy lost when dropping to 3 or 4 bits.

The intuition here is that LLMs are highly sensitive to activation outliers. Methods like SmoothQuant by Guangxuan Xiao mathematically migrate the "difficulty" of quantization from the volatile activations to the more stable weights. By combining this with learnable dequantization scales, as in TesseraQ, researchers can now push models like Llama-2 into the "ultra-low" 2-bit regime while maintaining usable performance.

Why It Works: The Unified Search Space

The success of these methods stems from treating quantization as a joint optimization problem. In the past, we optimized for accuracy, then compressed. Differentiable search frameworks like AutoQNN optimize for accuracy and hardware constraints simultaneously.

When the bit-width and the weights are in the same gradient-based search space, the model can make trade-offs that a human engineer never could: it might decide to use 6 bits for a critical attention head while dropping a redundant feed-forward layer to a mere 2 bits. This automated, fine-grained allocation of "bit budget" is what allows modern quantized models to achieve 30x compression with negligible loss in fidelity.

Papers in this version

Showing 60 of 292 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 227 papers
  2. Version 1 · 2026-10-04 · 292 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_learnable_quantization_scaling_f25b1413,
  title = {learnable quantization scaling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/learnable-quantization-scaling/rdr_f25b14132ab34536a1f17e0397661d57},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_differentiable_search_for_mixed_precisio_f25b1413_v1,
  title = {Differentiable Search for Mixed Precision Quantization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/differentiable-search-for-mixed-precision-quantization/rdr_f25b14132ab34536a1f17e0397661d57/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}