ATQ: Activation Transformation forWeight-Activation Quantization of Large Language Models

Yundong Gai, Ping LiEMNLP 2024 · OpenReview

Abstract

Deploying Large Language Models (LLMs) is hindered by immense computational and storage costs. While various quantization methods have emerged, their accuracy often falls short of academic and industrial requirements. We propose ATQ, an INT8 weight-activation quantization method that achieves near-lossless performance. By employing a mathematically equivalent transformation and applying the triangle inequality, we constrain quantization error to the sum of independent weight and activation errors. Transformed activations are normalized into a range suitable for quantization, while weight errors are compensated via optimization. Our experiments on OPT and LLaMA families demonstrate that ATQ maintains perplexity increases within 1.0 and limits accuracy degradation to under 0.5%, even in worst-case scenarios.

Introduction

The primary bottleneck in LLM deployment is the precision of activations. Unlike weights, activations often contain "outliers"—specific channels with magnitudes significantly higher than the rest. Standard 8-bit quantization (W8A8) typically scales the entire range to fit into 256 levels; these outliers force the remaining "normal" values into a tiny fraction of the available bit-depth, leading to catastrophic precision loss.

Existing solutions like SmoothQuant attempt to migrate this difficulty from activations to weights, but they often require manual hyperparameter tuning and struggle with larger models. ATQ addresses this by mathematically transforming the activation distribution to eliminate the impact of outliers before quantization occurs.

Methodology

ATQ utilizes a two-pronged strategy to decouple and minimize quantization error:

  1. Error Decomposition: Using the triangle inequality, the authors prove that the total layer error is bounded by the sum of the weight quantization error and the activation quantization error. This allows for independent optimization of both components.
  2. Activation Transformation (AT): Instead of quantizing raw activations, ATQ applies a linear transformation consisting of a Shift (centering the activation range per channel at zero) and a Scale (normalizing outliers to a standard range, typically $[-1, 1]$).
  3. Mathematical Equivalence: To preserve model logic, the inverse of this transformation is integrated into the weights and biases of the linear layers. This makes the process transparent to the model's inference while providing hardware-friendly INT8 activations.
  4. Weight Compensation (EC): The weights, which now incorporate the complexity of the activation transformation, are quantized using an optimized version of the GPTQ algorithm. This compensates for the shift by adjusting the remaining unquantized weights within each layer.

Experimental Results

Language Modeling Performance

ATQ was evaluated across the OPT and LLaMA model families. As shown in the perplexity results for LLaMA models, ATQ consistently matches the FP16 baseline (W16A16) and outperforms existing W8A8 methods. Notably, ATQ avoids the "Out of Memory" (OOM) issues encountered by competitors like SmoothQuant and OmniQuant when scaling to 70B parameter models.

Perplexity results on WikiText2 for LLaMA models. ATQ achieves near-baseline performance and avoids the memory issues seen in competing methods.

Zero-Shot Accuracy

To verify the "lossless" claim, the authors tested the models on a suite of zero-shot tasks, including PIQA, ARC, and HellaSwag. Across the OPT family, ATQ maintained accuracy within 0.5% of the original floating-point model. In many instances, ATQ's W8A8 performance was indistinguishable from the W8A16 (weight-only quantization) baseline, representing a significant step forward for full INT8 deployment.

Zero-shot accuracy benchmarks for the OPT model family. ATQ maintains performance within 0.5% of the FP16 baseline across multiple tasks.

Key Insights and Conclusion

ATQ demonstrates that activation outliers are primarily a range-management problem. By mathematically normalizing the activation range before quantization, the problem becomes significantly more tractable. Unlike gradient-based optimization methods (e.g., OmniQuant), ATQ relies on direct mathematical transformations, making it faster and more robust across different model architectures, including the recent Llama-3 family.

While current results are based on simulated quantization, the mathematical framework provides a clear path for hardware-level INT8 implementation. Future work will focus on developing optimized CUDA kernels to translate these accuracy gains into real-world inference speedups and exploring even lower bit-widths (e.g., W4A4).

Summary updated

Abstract

There are many emerging quantization methods to resolve the problem that the huge demand on computational and storage costs hinders the deployment of Large language models (LLMs). However, their accuracy performance still can not satisfy the entire academic and industry community. In this work, we propose ATQ, an INT8 weight-activation quantization of LLMs, that can achieve almost lossless accuracy. We employ a mathematically equivalent transformation and a triangle inequality to constrain weight-activation quantization error to the sum of a weight quantization error and an activation quantization error. For the weight part, transformed weights are quantized along the |in-feature| dimension and the quantization error is compensated by optimizing following in-features. For the activation part, transformed activations are in the normal range and can be quantized easily. We provide comparison experiments to demonstrate that our ATQ method can achieve almost lossless in accuracy on OPT and LLaMA families in W8A8 quantization settings. The increase of perplexity is within 1 and the accuracy degradation is within 0.5 percent even in the worst case.
BibTeX
@inproceedings{gai2024atq,
  author = {Yundong Gai and Ping Li},
  title = {{ATQ: Activation Transformation forWeight-Activation Quantization of Large Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=MJLE7vp5gB},
}
Figure