The Illusion of Weight-Only Compression
When post-training weight quantization methods like OPTQ: Accurate Quantization for Generative Pre-trained Transformers (developed by Elias Frantar et al.) and SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression (developed by Tim Dettmers et al.) emerged, they appeared to solve the deployment challenges of Large Language Models (LLMs). These techniques demonstrated that static model weights could be compressed down to 3 or even 2 bits per parameter with minimal loss in accuracy, drastically reducing the storage footprint and allowing massive models to fit onto consumer-grade GPUs.
However, compressing the weights addresses only the static portion of a model's memory footprint. In an active inference pipeline, the data flowing through the network—the activations—remains dynamic. By focusing exclusively on weights, traditional quantization strategies leave activations in high-precision formats like FP16 or BF16. This creates a severe operational imbalance, causing systems to hit two distinct limitations: a dynamic memory wall and a compute wall.
The Dynamic Memory Wall: The KV Cache Crisis
The primary reason weight-only quantization fails to scale with long sequences is the Key-Value (KV) cache. During autoregressive sequence generation, the model stores the key and value states of all previous tokens to avoid redundant computations at each new step. While the model weights remain constant in size, the KV cache grows linearly with both the batch size and the sequence length.
As noted by Louis Leconte et al. in ReALLM: A general framework for LLM compression and fine-tuning and Albert Tseng et al. in QuIP$\#$: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks, the memory consumed by these dynamic activation tensors can quickly rival or exceed the storage required for the model weights themselves during long-context inference. If a model's weights are compressed to 2 bits but its KV cache remains at 16 bits, the context window remains severely constrained.
To achieve comprehensive model reduction, compression frameworks must expand their scope to include dynamic states. For example, Abhishek Sharma et al. introduce MiniKV: Pushing the Limits of 2-Bit KV Cache via Compression and System Co-Design for Efficient Long Context Inference to compress the cache directly, while Minsu Kim 0004 et al. propose Oaken: Fast and Efficient LLM Serving with Online-Offline Hybrid KV Cache Quantization to optimize dynamic memory buffers. Both works emphasize that a holistic solution requires unifying weight compression with activation and cache quantization.
The Compute Wall: Why Weight-Only Quantization Limits Speed
The second limitation of weight-only quantization is computational. Modern hardware accelerators feature specialized, high-throughput integer arithmetic units (such as INT8 or INT4 Tensor Cores) designed to execute low-precision operations much faster than standard floating-point operations.
When a model uses weight-only quantization, the weights are stored in low precision but must be dequantized back into floating-point format on-the-fly before performing matrix multiplication with the high-precision activations. As observed by Hanlin Tang et al. in EasyQuant: An Efficient Data-free Quantization Algorithm for LLMs and Janghwan Lee et al. in Enhancing Computation Efficiency in Large Language Models through Weight and Activation Quantization, this requirement means the underlying mathematical operations cannot leverage low-precision hardware arithmetic units to accelerate execution speeds.
Furthermore, Yilong Zhao et al. demonstrate in Atom: Low-bit Quantization for Efficient and Accurate LLM Serving that as serving batch sizes increase, the system transitions from being memory-bound (limited by weight transfer bandwidth) to compute-bound (limited by raw arithmetic processing). In these compute-bound scenarios, weight-only quantization hits a performance ceiling. True hardware acceleration requires quantizing both weights and activations to unlock integer math pipelines.
Overcoming Activation Outliers
If quantizing activations is essential for both memory and speed, why has the field lagged in adopting it? The primary obstacle is the presence of activation outliers.
While weight distributions tend to be relatively smooth and uniform, activation distributions in modern Transformers exhibit extreme spikes. A tiny fraction of feature channels can have magnitudes that are orders of magnitude larger than the rest. If standard uniform linear quantization is applied across the entire activation tensor, these massive outliers force the quantization scale to be excessively wide. This squashes the remaining 99% of normal values into just a few quantization bins, destroying the model's representational capacity.
To resolve this activation precision bottleneck, researchers have developed outlier-aware and error-correcting architectures:
- Outlier Isolation: Changhun Lee et al. propose OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Models, which identifies the specific weight columns corresponding to activation outliers and preserves them in high precision. For joint weight-activation quantization, Ilia Markov introduces QUIK: Towards End-to-end 4-Bit Inference on Generative Large Language Models, a hybrid scheme that keeps a tiny fraction of outlier features in higher precision while executing the rest in 4-bit, enabling practical end-to-end low-bit inference.
- Error Correction: Rather than avoiding outliers, some methods allow quantization noise to occur and then correct it mathematically. Meyer Scetbon et al. introduce Low-Rank Correction for Quantized LLMs, which adds small, full-precision "sidecar" matrices to each layer to correct the systematic errors introduced by aggressive activation quantization.
By shifting the focus from static weight reduction to a unified treatment of weights, activations, and caches, these strategies bridge the activation quantization gap, moving the field past simple storage compression toward full-stack hardware acceleration.