Zero-Shot Dynamic Quantization for Transformer Inference

Yousef El-Kurdi, Jerry Quinn, Avi SilEMNLP 2022 · DOI

Abstract

Large-scale Transformer models are the standard for modern Natural Language Processing (NLP), yet their computational demands pose significant challenges for deployment on resource-constrained hardware. While 8-bit integer (INT8) quantization offers a path toward 4x inference speedups and reduced memory footprints, it often results in substantial accuracy degradation due to activation outliers. This paper introduces TM-IQR (Token-Maximums Interquartile Range), a novel "plug-and-play" dynamic quantization method. TM-IQR requires zero retraining and no calibration datasets, making it a truly zero-shot solution. By utilizing robust statistics to identify and clip outliers at runtime, TM-IQR recovers nearly all accuracy lost during standard quantization with less than a 2% latency overhead.

Introduction

The transition from 32-bit floating-point (FP32) weights and activations to 8-bit integers is a critical step for optimizing Transformers for CPUs and mobile devices. However, the standard approach to quantization often faces a trade-off between performance and complexity. Existing solutions typically rely on two methods:

  1. Quantization-Aware Training (QAT): Requires retraining the model with the original data, which is computationally expensive and often impossible for proprietary or "black-box" models.
  2. Post-Training Calibration: Requires a representative "held-out" dataset to determine clipping thresholds, which may not be available in privacy-sensitive environments.

TM-IQR addresses these limitations by providing a data-free, runtime strategy that calculates optimal quantization scales on-the-fly.

Methodology: Addressing the Outlier Problem

The primary cause of accuracy loss in INT8 quantization is the presence of outliers—activations with extreme values that expand the quantization range. When the range is too wide, the majority of "normal" values are compressed into a very small number of integer bins, leading to a loss of precision known as "swamping."

The TM-IQR Solution

The authors propose using the Interquartile Range (IQR), a statistical measure of variability based on dividing a data set into quartiles. Unlike mean or variance, IQR has a high "breakdown point" (25%), meaning it remains stable even if a significant portion of the data consists of extreme outliers.

To implement this efficiently during inference, the authors developed Token-Maximums IQR. A standard IQR calculation requires sorting every element in a tensor, which is computationally prohibitive ($O(N \log N)$). TM-IQR optimizes this by:

  1. Calculating the maximum absolute value for each token in a sequence.
  2. Sorting only these token-level maximums.
  3. Determining the clipping threshold based on the IQR of these maximums.

This reduces the complexity to $O(N + L \log L)$, where $L$ is the sequence length, ensuring the method is fast enough for real-time applications.

Targeted Application

Through error analysis, the researchers identified the input to the second feed-forward layer (FF2) in the Transformer block as the most significant source of quantization error. This layer typically features wide dimensions and unbounded activation functions (e.g., ReLU or GELU). By applying TM-IQR specifically at this bottleneck, the model achieves maximum accuracy recovery with minimal impact on throughput.

Experimental Evaluation

The effectiveness of TM-IQR was evaluated across several diverse NLP benchmarks and architectures, including BERT, RoBERTa, and XLM-R.

Performance on GLUE

On the GLUE benchmark (including tasks like SST-2, CoLA, and MNLI), standard 8-bit quantization significantly degraded performance. For RoBERTa-base, the average score dropped from an FP32 baseline of 82.0 to 75.2. Applying TM-IQR recovered the score to 80.8, bridging the majority of the gap without any model fine-tuning.

Multilingual and Question Answering

The method proved equally robust in multilingual settings. On the TyDI benchmark (covering 11 diverse languages) using XLM-R-base, TM-IQR achieved a score of 67.0, nearly matching the original FP32 performance of 67.7, while standard INT8 quantization lagged at 62.9.

Computational Efficiency

A critical requirement for dynamic quantization is low latency. The authors demonstrated that the overhead of calculating TM-IQR at runtime results in a throughput decrease of less than 2% compared to naive 8-bit quantization, making it highly viable for production environments.

Conclusion

TM-IQR represents a significant advancement in zero-shot model compression. By leveraging robust statistics and optimizing for runtime complexity, it allows for the deployment of high-performance Transformer models in INT8 mode without the need for original training data or calibration sets. Future work may explore the application of this statistical clipping approach to generative decoder models (like GPT) and lower bit-depths (4-bit or 2-bit quantization).

Summary updated

BibTeX
@inproceedings{elkurdi2022zero,
  author = {Yousef El-Kurdi and Jerry Quinn and Avi Sil},
  title = {{Zero-Shot Dynamic Quantization for Transformer Inference}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-industry.45},
}
Figure