Understanding and Overcoming the Challenges of Efficient Transformer Quantization

Yelysei Bondarenko, Markus Nagel, Tijmen Blankevoort

Abstract

Transformer-based architectures have become the de-facto standard models for a wide range of Natural Language Processing tasks. However, their memory footprint and high latency are prohibitive for efficient deployment and inference on resource-limited devices. In this work, we explore quantization for transformers. We show that transformers have unique quantization challenges -- namely, high dynamic activation ranges that are difficult to represent with a low bit fixed-point format. We establish that these activations contain structured outliers in the residual connections that encourage specific attention patterns, such as attending to the special separator token. To combat these challenges, we present three solutions based on post-training quantization and quantization-aware training, each with a different set of compromises for accuracy, model size, and ease of use. In particular, we introduce a novel quantization scheme -- per-embedding-group quantization. We demonstrate the effectiveness of our methods on the GLUE benchmark using BERT, establishing state-of-the-art results for post-training quantization. Finally, we show that transformer weights and embeddings can be quantized to ultra-low bit-widths, leading to significant memory savings with a minimum accuracy loss.

Introduction

Transformer models like BERT and GPT have revolutionized natural language processing but are computationally expensive. Quantization—the process of converting 32-bit floating-point (FP32) values to lower-bit integers (e.g., INT8)—is a standard technique to reduce memory and latency. However, while quantization is effective for computer vision, it often causes a catastrophic drop in Transformer accuracy. This work investigates the root causes of this failure and proposes novel solutions to enable efficient 8-bit and ultra-low-bit inference.

Identifying the Quantization Bottleneck

Our investigation into BERT reveals that the primary source of quantization error is not the weights, but the activations. Specifically, the residual connections following the Feed-Forward Networks (FFN) are highly sensitive.

Sensitivity analysis of quantized activations in BERT. The table shows that keeping residual connections after FFN in higher precision significantly recovers performance across GLUE tasks.

We discovered that these activations contain "structured outliers"—values in specific embedding dimensions that are significantly larger than the rest. These outliers are not random noise; they play a functional role in helping the attention mechanism focus on special tokens like [SEP]. Standard quantization "squishes" the precision of normal values to accommodate these outliers, destroying the model's reasoning capabilities.

Visualization of structured outliers in activations. (a) shows high dynamic ranges for specific tokens, while (b) shows these outliers are concentrated in specific embedding dimensions.

Proposed Solutions

To address these challenges, we propose three distinct strategies:

  1. Mixed-Precision PTQ: By identifying and keeping only the most sensitive tensors (roughly 22%) in 16-bit while quantizing the rest to 8-bit, we can recover most of the original performance.
  2. Per-Embedding-Group (PEG) Quantization: This novel scheme groups embedding dimensions. Since outliers are dimension-specific, providing each group with its own scaling factor prevents outliers from degrading the precision of other values.
  3. Quantization-Aware Training (QAT): For maximum accuracy, we use fine-tuning with learnable quantization ranges, allowing the model to adapt to the noise introduced by low-bit representations.

Experimental Results

We evaluated our methods on the GLUE benchmark. Standard 8-bit quantization (W8A8) on the MNLI task causes BERT's score to drop from 84.91 to 50.31. Our PEG-PTQ method recovers this to 81.25 without retraining. Our mixed-precision approach achieves a GLUE score of 82.43, outperforming previous specialized methods. Furthermore, combining QAT with ultra-low bit-widths (4-bit weights, 2-bit embeddings) achieves an 8.85× reduction in model size with less than 1% accuracy loss.

Conclusion

The deployment of Transformers on edge devices is hindered by structured outliers in activations. By pinpointing these bottlenecks and employing targeted quantization schemes like PEG or mixed precision, we can achieve significant compression and efficiency gains with minimal impact on model performance.

Summary updated

Abstract

Transformer-based architectures have become the de-facto standard models for a wide range of Natural Language Processing tasks. However, their memory footprint and high latency are prohibitive for efficient deployment and inference on resource-limited devices. In this work, we explore quantization for transformers. We show that transformers have unique quantization challenges -- namely, high dynamic activation ranges that are difficult to represent with a low bit fixed-point format. We establish that these activations contain structured outliers in the residual connections that encourage specific attention patterns, such as attending to the special separator token. To combat these challenges, we present three solutions based on post-training quantization and quantization-aware training, each with a different set of compromises for accuracy, model size, and ease of use. In particular, we introduce a novel quantization scheme -- per-embedding-group quantization. We demonstrate the effectiveness of our methods on the GLUE benchmark using BERT, establishing state-of-the-art results for post-training quantization. Finally, we show that transformer weights and embeddings can be quantized to ultra-low bit-widths, leading to significant memory savings with a minimum accuracy loss. Our source code is available at~\url{https://github.com/qualcomm-ai-research/transformer-quantization}.
BibTeX
@inproceedings{bondarenko2021understanding,
  author = {Yelysei Bondarenko and Markus Nagel and Tijmen Blankevoort},
  title = {{Understanding and Overcoming the Challenges of Efficient Transformer Quantization}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=laOElRC1Um},
}
Figure