Towards Efficient Post-training Quantization of Pre-trained Language Models

Haoli Bai, Lu Hou, Lifeng Shang, Xin Jiang, Irwin King, Michael Lyu

TL;DR

Researchers have developed a new post-training quantization (PTQ) method called MREM that makes compressing large language models significantly faster and less resource-intensive. By optimizing the model in "modules" rather than individual layers and using a clever parallel training strategy, they achieved performance close to full quantization-aware training while being 150x faster and using only one-third of the memory.

Radar chart comparing MREM and QAT across Accuracy, Training Time, Data, and Memory

What Problem Does This Paper Address?

Large Pre-trained Language Models (PLMs) like BERT are powerful but massive, making them difficult to deploy on edge devices. To shrink them, researchers use quantization—converting high-precision numbers (32-bit) into low-precision ones (like 2-bit or 4-bit).

Currently, the gold standard is Quantization-Aware Training (QAT), which retrains the whole model. However, QAT is a "resource hog": it requires the full original dataset, massive GPU memory, and days of training. The alternative, Post-Training Quantization (PTQ), is much lighter but usually results in a massive drop in accuracy because it typically optimizes each layer in isolation, ignoring how errors accumulate across the network.

Methods

The authors introduce Module-wise Reconstruction Error Minimization (MREM). Instead of looking at one layer at a time, they group several Transformer layers into a "module" and minimize the difference between the output of the original full-precision module and the new quantized version.

To make this even faster, they implemented two key innovations:

  1. Accelerated Parallel Training: They split the model into modules and put each module on a different GPU. Using an "input queue" system (stale synchronous parallel), all GPUs can work simultaneously without waiting for the previous one to finish a full pass.
  2. Annealed Teacher Forcing: Parallel training can be unstable because a module might be learning from the "noisy" (unoptimized) output of the previous quantized module. The authors use "Teacher Forcing"—initially feeding the quantized module clean signals from the full-precision model and gradually "annealing" (reducing) that help until the module learns to handle its own quantized inputs.

Datasets

The researchers evaluated their method on standard Natural Language Understanding (NLU) benchmarks:

  • GLUE (General Language Understanding Evaluation): Includes tasks like MNLI, QQP, QNLI, and SST-2.
  • SQuAD (Stanford Question Answering Dataset): Versions v1.1 and v2.0 for reading comprehension.

Crucially, while QAT uses the entire dataset (hundreds of thousands of samples), MREM only requires a tiny "calibration set" of about 1,000 to 4,000 random samples.

Key Insights and Ideas

  • Granularity Matters: Layer-wise optimization is too narrow (errors pile up), and end-to-end training is too heavy. Module-wise optimization is the "Goldilocks" zone—it captures enough inter-layer correlation to maintain accuracy without the overhead of full retraining.
  • Breaking the Sequential Bottleneck: Traditionally, PTQ is sequential (you must finish layer 1 to start layer 2). Their parallel strategy with input queues effectively turns a sequential process into a simultaneous one, approaching theoretical linear speedup (e.g., 4x faster on 4 GPUs).
  • Data Efficiency: The "aha moment" is that you don't need a massive dataset to fix quantization errors. By focusing on reconstructing the output of the original model, a few thousand samples provide enough signal to calibrate the weights.

Key Results

  • Speed: For a BERT-large model, MREM is over 150x faster than traditional QAT.
  • Memory: It uses roughly 1/3 of the GPU memory compared to QAT, allowing researchers to quantize large models on consumer-grade hardware (like a single GTX 1080 Ti).
  • Accuracy: On the MNLI dataset, a 4-bit BERT-large model using MREM achieved 85.5% accuracy, which is nearly identical to the full-precision model and significantly better than previous PTQ methods (which often dropped to ~70%).
  • Extreme Compression: Even at 2-bit weights, the model retained a high level of performance, outperforming other state-of-the-art PTQ methods like BRECQ and QDrop.
Detailed comparison of Training Time, Memory, Data, and Performance

Limitations and Future Work

  • Architecture Scope: The paper focuses heavily on BERT-style Transformer models. While the logic should hold for newer architectures like GPT or Llama, those specific experiments aren't detailed here.
  • Hardware Dependency: While the method reduces memory, the parallel speedup requires multiple GPUs. For users with only one GPU, the "parallel" benefit is lost, though the "module-wise" accuracy benefit remains.
  • Future Directions: The authors suggest exploring this method for even larger generative models (LLMs) with billions of parameters, where the memory savings would be even more critical.

Summary updated

Abstract

Network quantization has gained increasing attention with the rapid growth of large pre-trained language models~(PLMs). However, most existing quantization methods for PLMs follow quantization-aware training~(QAT) that requires end-to-end training with full access to the entire dataset. Therefore, they suffer from slow training, large memory overhead, and data accessibility issues. In this paper, we study post-training quantization~(PTQ) of PLMs, and propose module-wise quantization error minimization~(MREM), an efficient solution to mitigate these issues. By partitioning the PLM into multiple modules, we minimize the reconstruction error incurred by quantization for each module. In addition, we design a new model parallel training strategy such that each module can be trained locally on separate computing devices without waiting for preceding modules, which brings nearly the theoretical training speed-up (e.g., $4\times$ on $4$ GPUs). Experiments on GLUE and SQuAD benchmarks show that our proposed PTQ solution not only performs close to QAT, but also enjoys significant reductions in training time, memory overhead, and data consumption.
BibTeX
@inproceedings{bai2022towards,
  author = {Haoli Bai and Lu Hou and Lifeng Shang and Xin Jiang and Irwin King and Michael Lyu},
  title = {{Towards Efficient Post-training Quantization of Pre-trained Language Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=tvDRmAxGIjw},
}
Figure