Standard neural network quantization treats all weights equally. When compressing a weight matrix $W$ to a lower-bit representation $\hat{W}$, traditional methods focus on minimizing the weight reconstruction error:
$$\arg\min_{\hat{W}} \|W - \hat{W}\|^2$$
However, a neural network never encounters weights in isolation. During a forward pass, weights are always multiplied by incoming activations ($X$). The true error introduced into the network's internal representations is not the weight error, but the output signal error:
$$\arg\min_{\hat{W}} \|WX - \hat{W}X\|^2$$
This distinction is the core insight behind activation-aware codebook optimization. If a weight is multiplied by an activation value that is consistently near zero, even a massive quantization error on that weight will have virtually no impact on the model's output. Conversely, if a weight interacts with a massive outlier activation—a common phenomenon in Large Language Models (LLMs)—even a tiny quantization error can scale up, corrupting the signal and derailing downstream performance.
By shifting the optimization objective from weight space to activation space, recent methods achieve ultra-low bit-widths (down to 1.61 or 2 bits) with negligible loss in accuracy.
Shifting the Objective to Output Signals
The foundational premise of this paradigm is that preserving the network's output signal matters far more than preserving the exact original values of the weights. In early work, Stock et al. demonstrated this in And the Bit Goes Down: Revisiting the Quantization of Neural Networks by weighting the reconstruction error of vector quantization according to the importance of input activations. This allowed them to compress a ResNet-50 down to just 5MB while maintaining high accuracy on ImageNet, proving that data distributions must guide the compression process.
When we look at modern architectures like LLMs, this principle becomes even more critical. The optimization process naturally protects the network's most influential data pathways by ensuring that the quantization codebook—the set of discrete representative values used to approximate the continuous weights—is shaped by the scale of the data flowing through it.
Operationalizing with Weighted Clustering
To build quantization codebooks that respect activation magnitudes, researchers modify traditional clustering algorithms like K-Means or Product Quantization. Instead of treating every weight vector as an equally important point in space, the clustering loss scales each weight's contribution based on its corresponding activation statistics.
Standard K-Means: [Weight Point] --------> Equal Importance
Weighted K-Means: [Weight Point] x (Activation Magnitude)² --------> Saliency-Driven Centroids
Several recent papers demonstrate different ways to compute and apply these activation weights:
- Average Magnitudes: In any4: Learned 4-bit Numeric Representation for LLMs, Elhoushi et al. modify the clustering objective to weight each weight's importance by the average magnitude of its corresponding activations. This ensures that the learned 4-bit codebook values are allocated more densely around regions that handle highly influential signals.
- Forward Statistics Only: IslamBouli et al. show in AAAC: Activation-Aware Adaptive Codebooks for 4-bit LLM Weight Quantization that simple forward-pass activation statistics provide enough information to optimize adaptive codebooks. This local approach matches the performance of heavy gradient-based methods without requiring expensive backpropagation.
- Squared Calibration Values: For architectures that rely heavily on element-wise multiplications, such as the RWKV family, standard quantization often fails. To solve this, Yue et al. in RWKVQuant: Quantizing the RWKV Family with Proxy Guided Hybrid of Scalar and Vector Quantization and Xu et al. in RWKVQuant: Quantizing the RWKV Family with Proxy Guided Hybrid of Scalar and Vector Quantization incorporate the squared values of calibration activations into the codebook generation process. This specific weighting preserves the integrity of non-linear fusion paths where errors would otherwise compound exponentially.
- Second-Moment Shaping: In the NoWag framework, presented by Liu et al. in both NoWag: A Unified Framework for Shape Preserving Compression of Large Language Models and NoWag: A Unified Framework for Shape Preserving Com- pression of Large Language Models, vector quantization is driven by the second moment of sample activations. This shape-preserving approach allows for efficient compression with drastically reduced calibration data overhead.
Defining Saliency: Magnitudes vs. Gradients
While activation magnitude is a powerful and computationally cheap proxy for weight importance, it does have a blind spot: high-magnitude activations can sometimes represent systematic bias or noise rather than useful task signal. To address this, some frameworks incorporate gradient information to measure how sensitive the final task loss actually is to specific activation channels.
In Inlier-Centric Post-Training Quantization for Object Detection Models, Kim et al. argue that gradient sensitivity serves as a more reliable indicator of feature importance than simple magnitude. By identifying which activations directly influence the task loss, their method prioritizes preserving the signal that contributes most to downstream accuracy.
Similarly, Xiao et al. introduce Task-Circuit Quantization (TACQ) in Task-Circuit Quantization: Leveraging Knowledge Localization and Interpretability for Compression. TACQ combines Quantization-Aware Localization (QAL)—which simulates quantization to estimate loss impact—with a Magnitude-Sharpened Gradient (MSG). This hybrid metric balances the identification of task-specific "circuit" weights with generally important weights, preventing the oversight of critical parameters at ultra-low bit-widths (2-3 bits).
Structural and Dynamic Adaptations
Activation awareness can also be applied structurally to minimize metadata overhead, or dynamically to adapt to real-time data shifts.
Because activations often exhibit clear channel-wise patterns, the quantization scheme can be structurally aligned to these patterns. In PTQ1.61: Push the Real Limit of Extremely Low-Bit Post-Training Quantization Methods for Large Language Models, Zhao et al. show that quantization error is primarily driven by these structured activation patterns rather than random weight distributions. By aligning a 1D structured quantization mask with these activation channels, they retain critical information at an extreme 1.61-bit compression level with negligible metadata overhead.
Alternatively, weights can be organized to accommodate sensitivity before quantization occurs. In Dual Precision Quantization for Efficient and Accurate Deep Neural Networks Inference, Karnieli et al. propose a Group-Aware Reordering strategy. Weights are statically reordered within local groups based on their sensitivity before quantization begins, allowing the most critical weights to be represented with higher accuracy without requiring complex dynamic indexing during inference.
Finally, activation-aware quantization can move entirely into the inference loop. In TTQ: Activation-Aware Test-Time Quantization to Accelerate LLM Inference On The Fly, Koike-Akino et al. introduce a method that quantizes weights on the fly during inference. By adapting the quantization scheme to the exact activation statistics of the current prompt, they eliminate the need for static calibration data and completely bypass the domain shift issues that plague offline post-training quantization.
Key Foundational Work
RWKVQuant: Quantizing the RWKV Family with Proxy Guided Hybrid of Scalar and Vector Quantization (2025) — contributed 2 concept elements to this direction.
RWKVQuant: Quantizing the RWKV Family with Proxy Guided Hybrid of Scalar and Vector Quantization (2025) — contributed 2 concept elements to this direction.
any4: Learned 4-bit Numeric Representation for LLMs (2025) — contributed 2 concept elements to this direction.