Joint Weight and Activation Quantization for LLM Inference

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2411 papers431 conceptsfrozen 2026-10-04

In the race to deploy Large Language Models (LLMs) on consumer hardware, researchers first targeted the most obvious bottleneck: the massive weight matrices. Techniques like GPTQ and AWQ successfully compressed 16-bit weights down to 4 or even 2 bits, allowing models with tens of billions of parameters to fit into a single GPU's VRAM.

However, a new consensus has emerged across recent research: weight-only quantization is hitting a point of diminishing returns. As Sehoon Kim and colleagues argue in SqueezeLLM, shrinking weights alone fails to address the "Memory Wall" created by dynamic activations and the Key-Value (KV) cache. To achieve true system-wide efficiency, we must move toward Full System Quantization—the joint compression of weights, activations, and the KV cache.

The Shifting Bottleneck

To understand why weight-only quantization is insufficient, we must look at how LLMs use memory during inference. Inference typically happens in two phases: the prefill phase (processing the input prompt) and the decoding phase (generating tokens one by one).

In the decoding phase, the system is often "memory-bound"—the speed is limited by how fast we can move weights from VRAM to the processor. Weight-only quantization helps here by reducing the data movement. But as we generate longer sequences, the KV cache (which stores previous context) grows linearly. For long-context tasks, Baohao Liao notes in ClusComp that high-precision activations can trigger out-of-memory errors even if the weights are tiny.

Furthermore, weight-only quantization creates a "compute mismatch." If weights are 4-bit but activations remain 16-bit, the hardware must dequantize the weights back to 16-bit before performing math. As Yilong Zhao and the Atom team point out, this prevents the system from using high-throughput integer arithmetic units (like INT8 or INT4 Tensor Cores), which are significantly faster than floating-point units.

Managing the Sensitivity of Activations

The primary reason activations are rarely quantized is their volatility. Unlike weights, which are static after training, activations change with every input. They often contain "outliers"—specific channels with magnitudes significantly higher than the rest—that, if crushed by quantization, destroy the model's reasoning capabilities.

Several strategies have emerged to handle this:

  1. Outlier Isolation: Tim Dettmers and Elias Frantar introduced SpQR, which identifies a tiny fraction of "salient" weights and activations to keep in high precision while quantizing the rest. Similarly, EasyQuant isolates the top 1% of outliers to maintain accuracy without needing calibration data.
  2. Activation-Aware Weighting: Instead of treating weights and activations as separate problems, AWQ (by Chuang Gan and Song Han) protects weights that correspond to high-magnitude activation channels. This "activation-aware" approach ensures that the most important data paths remain precise.
  3. Error Correction: Meyer Scetbon and James Hensman proposed Low-Rank Correction (LRC), which adds small "sidecar" matrices to each layer. These matrices are trained to "fix" the noise introduced when both weights and activations are pushed to 4-bit (W4A4), closing the performance gap by over 50%.

System and Hardware Co-Design

The ultimate goal of this research direction is a unified inference pipeline where every component is compressed. MiniKV demonstrates this by combining 2-bit KV cache quantization with "pyramid" token eviction, drastically reducing the memory footprint for long-context tasks.

When weights and activations are both quantized, the system can finally move from being memory-bound to leveraging the full computational power of the chip. Atom achieves a 7.7x throughput improvement by using 4-bit weight-activation quantization and high-performance fused kernels. This shift is essential for "cloud-scale" production, as seen in the deployment of massive Mixture of Experts (MoE) models Who Says Elephants Can’t Run, where 4-bit quantization and optimized kernels enabled a 26x throughput boost.

By treating the model not as a collection of static weights, but as a dynamic flow of data, researchers are finally breaking through the memory wall, enabling larger models to run faster on smaller, more accessible hardware.

Papers in this version

Showing 60 of 411 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 335 papers
  2. Version 1 · 2026-10-04 · 411 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_activation_quantization_gap_1c631d05,
  title = {activation quantization gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/activation-quantization-gap/rdr_1c631d05926a4ea5bcad8141f6484888},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_joint_weight_and_activation_quantization_1c631d05_v1,
  title = {Joint Weight and Activation Quantization for LLM Inference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/joint-weight-and-activation-quantization-for-llm-inference/rdr_1c631d05926a4ea5bcad8141f6484888/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}