Any-Bit Quantization for Fine-Grained Model Compression

This direction is split into other directions: fractional bit-width quantization. Its last version stays readable and citable here.

Version 1 of 150 papers51 conceptsfrozen 2026-10-04

In the standard deployment pipeline, compressing a Large Language Model (LLM) feels like trying to fit a square peg into a round hole. If a 4-bit model is too large for a 24GB GPU, but a 2-bit model is too "dumb" to be useful, practitioners are left with a massive performance gap and no middle ground. This "integer bit-width barrier" forces a choice between underutilizing hardware or sacrificing model intelligence.

A growing body of research is moving toward Any-bit Quantization, a paradigm that enables fractional bit-widths (e.g., 2.2-bit or 3.32-bit) and elastic memory footprints. By treating quantization as a continuous optimization problem rather than a discrete rounding exercise, these methods allow models to be "right-sized" to fit specific hardware constraints with megabyte-level precision.

The Integer Bit-Width Barrier

Traditional quantization maps high-precision weights to a fixed set of integers (e.g., -128 to 127 for 8-bit). This approach is rigid. As noted by Bo Liu and Qiang Liu in SKIM, restricting models to integer bit-widths prevents them from being precisely tailored to hardware memory. If your model's ideal size falls between 2 and 3 bits, you are forced to round down, often leading to a "performance collapse."

Furthermore, the math of low-bit quantization is often unnecessarily constrained. Lin Li and Xinyu Hu argue in NeUQI that forcing the "zero-point" (the value representing zero in the quantized space) to be an integer is a relic of old hardware that unnecessarily restricts the optimization search space, particularly in 2-bit and 3-bit regimes.

Achieving Fractional Precision

To break the integer barrier, researchers use several "lifting" and "mixing" mechanisms to achieve non-integer average bit-widths.

Channel-Wise Mixed Precision

Instead of quantizing the entire model to 3 bits, Zihan Chen and Cong Shen propose CMPQ (Channel-Wise Mixed-Precision Quantization). By identifying which weight channels are most sensitive to activation outliers, they can allocate 4 bits to critical channels and 2 bits to others. This results in a "fractional" average (e.g., 2.2 bits) that maximizes performance within a strict memory budget.

Dimensional Lifting and Frames

Another approach involves changing the space in which the weights live. Liulu He and colleagues introduced LiftQuant (Continuous Bit-Width LLM via Dimensional Lifting and Projection), which uses a "lift-then-project" mechanism to reach arbitrary bit-depths. Similarly, Harshavardhan Adepu and Vikas Singh developed FrameQuant (Flexible Low-Bit Quantization for Transformers), which maps weights into a redundant "Fusion Frame" space. This redundancy makes the model naturally robust to quantization noise, allowing for smooth transitions between bit-rates.

Residuals and Structural Approximation

If rounding a weight to the nearest integer creates an error, why not just quantize the error itself? This is the intuition behind Residual Error Expansion (REx), developed by Edouard Yvinec and Arnaud Dapogny (REx: Data-Free Residual Quantization Error Expansion). REx iteratively quantizes the "leftover" residuals, allowing developers to trade a small amount of extra computation for a massive jump in accuracy.

A similar concept appears in Network Sketching by Steven Y. Guo and Anbang Yao (Network Sketching: Exploiting Binary Structure in Deep CNNs). Instead of approximating values, they approximate the filter as a weighted sum of binary bases. By adding or removing these binary "sketches," they can scale model complexity with granular control. This structural approach is echoed in Group-Net (Structured Binary Neural Networks) by Bohan Zhuang and Chunhua Shen, which uses parallel binary branches to reconstruct the representational power of full-precision models.

Decoupling Storage from Precision

One of the most powerful shifts in this field is the use of Entropy Coding to decouple the physical storage size of a model from its numerical precision.

In CERWU (Reducing Storage of Pretrained Neural Networks by Rate-Constrained Quantization and Entropy Coding), Alexander Conzelmann and Robert Bamler integrate a rate-estimation term directly into the weight updates. This allows the model to learn weights that are not just accurate, but also highly compressible. Patrick Putzky and Thomas Wollmann take this further with EntQuant (Float8@2bits), showing that you can maintain high-precision formats (like Float8) for computation while using entropy coding to shrink the storage footprint to an average of 2 bits per parameter.

Elasticity: Models that Breathe

The ultimate goal of any-bit quantization is "elasticity"—the ability for a model to adapt its size dynamically.

By moving away from rigid bit-widths, this research direction ensures that no byte of hardware memory is wasted and no ounce of model performance is unnecessarily sacrificed.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_any_bit_quantization_for_fine_grained_mo_6a55f4bb,
  title = {Any-Bit Quantization for Fine-Grained Model Compression},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/any-bit-quantization-for-fine-grained-model-compression/rdr_6a55f4bb8e5e4f7e8f17e853032272e5},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_any_bit_quantization_for_fine_grained_mo_6a55f4bb_v1,
  title = {Any-Bit Quantization for Fine-Grained Model Compression},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/any-bit-quantization-for-fine-grained-model-compression/rdr_6a55f4bb8e5e4f7e8f17e853032272e5/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}