XTC: Extreme Compression for Pre-trained Transformers Made Simple and Efficient

Xiaoxia Wu, Zhewei Yao, Minjia Zhang, Conglong Li, Yuxiong HeNeurIPS 2022 · OpenReview

TL;DR

XTC (Extreme Transformer Compression) proves that we don't need complex, multi-stage pipelines to achieve massive model compression. By simply training longer with data augmentation and smart layer selection, the authors achieve a 50x reduction in model size while outperforming previous state-of-the-art methods that relied on expensive pre-training and intricate distillation schemes.

The Challenge of Extreme Compression

As Large Language Models (LLMs) grow to hundreds of billions of parameters, deploying them on edge devices (like smartphones and IoT hardware) becomes nearly impossible. Extreme compression—specifically quantizing weights down to 1-bit (binary) or 2-bit (ternary) precision—is the "holy grail" of efficiency, potentially offering 16x to 32x memory savings.

However, previous research suggested that 1-bit quantization causes a "sharp performance drop" compared to 2-bit. To fix this, researchers developed incredibly complex pipelines involving multi-stage knowledge distillation (KD), weight-splitting, and expensive pre-training. These methods are difficult to tune and computationally heavy. XTC asks a fundamental question: Are these complex tricks actually necessary, or are we just not training these tiny models correctly?

The XTC Method: Simplicity Wins

The authors propose a streamlined, two-step pipeline called XTC:

  1. Lightweight Layer Reduction: Instead of using computationally expensive pre-distilled models (like TinyBERT), they simply select a subset of layers from a fine-tuned teacher model (e.g., taking every other layer to create a "Skip-BERT").
  2. Robust 1-bit/2-bit Quantization: They apply Quantization-Aware Training (QAT) using a simple One-Stage Knowledge Distillation (1S-KD).

The "Secret Sauce"

The authors discovered that the perceived "failure" of 1-bit models was largely due to under-training. XTC succeeds by utilizing:

  • Longer training budgets: Giving the model more time to navigate the irregular loss landscape of binary weights.
  • Data Augmentation (DA): Using text-editing techniques to expand the training set, which is vital for preventing overfitting in extreme compression.
  • Learning Rate Decay: Carefully searching for optimal peak learning rates rather than relying on complex schedules.

Key Insights and Ideas

The "Under-training" Revelation

Previous baselines for 1-bit models were significantly under-trained. By simply increasing the training iterations (moving from a standard "Budget-A" to an extended "Budget-C"), a 1-bit model's performance gap compared to a 2-bit model almost entirely disappears.

Impact of increased training budgets on 1-bit and 2-bit model performance across GLUE tasks.

Pre-training is Optional

You don't necessarily need to perform expensive distillation during the pre-training phase. Initializing a student model with a subset of fine-tuned teacher weights (layer pruning) is surprisingly effective and much cheaper. Furthermore, multi-stage distillation (distilling hidden states first, then logits) is not strictly necessary. A well-tuned single-stage distillation can match or exceed the performance of complex multi-stage pipelines.

Data Augmentation is Non-negotiable

For extreme compression, Data Augmentation is critical. Without it, performance on smaller tasks like CoLA or RTE drops significantly, regardless of the distillation strategy used.

Key Results

Evaluated on the GLUE (General Language Understanding Evaluation) benchmark, XTC demonstrated remarkable efficiency and performance:

  • 50x Compression: By combining layer reduction (reducing 12 layers to 5 or 6) with 1-bit quantization, XTC reduces the model size by 50x while maintaining high accuracy on GLUE tasks.
  • Outperforming SOTA: A 5-layer XTC-BERT (initialized without expensive pre-training) outperformed the 6-layer TinyBERT, which requires massive pre-training resources.
  • 1-bit Breakthrough: XTC achieved a GLUE score of 83.44 with a 1-bit BERT-base, significantly higher than previous 1-bit methods like BinaryBERT (82.29) and much closer to the full-precision teacher (83.95).
GLUE score versus model size, demonstrating XTC's superior compression-to-performance ratio compared to SOTA methods.

Limitations and Future Work

While layer reduction (depth) worked exceptionally well, the authors noted that reducing the width (hidden dimension) of the model via simple weight selection caused a sharp accuracy drop. They suggest Neural Architecture Search (NAS) might be needed for width-based compression.

Additionally, this study focused primarily on weight quantization. Exploring the quantization of LayerNorm and activations in conjunction with XTC, as well as applying these findings to generative decoder models like GPT-2/3, remain exciting areas for future research.

Summary updated

Abstract

Extreme compression, particularly ultra-low bit precision (binary/ternary) quantization, has been proposed to fit large NLP models on resource-constraint devices. However, to preserve the accuracy for such aggressive compression schemes, cutting-edge methods usually introduce complicated compression pipelines, e.g., multi-stage expensive knowledge distillation with extensive hyperparameter tuning. Also, they oftentimes focus less on smaller transformer models that have already been heavily compressed via knowledge distillation and lack a systematic study to show the effectiveness of their methods. In this paper, we perform a very comprehensive systematic study to measure the impact of many key hyperparameters and training strategies from previous. As a result, we find out that previous baselines for ultra-low bit precision quantization are significantly under-trained. Based on our study, we propose a simple yet effective compression pipeline for extreme compression. Our simplified pipeline demonstrates that (1) we can skip the pre-training knowledge distillation to obtain a 5-layer \bert while achieving better performance than previous state-of-the-art methods, like TinyBERT; (2) extreme quantization plus layer reduction is able to reduce the model size by 50x, resulting in new state-of-the-art results on GLUE tasks.
BibTeX
@inproceedings{wu2022xtc,
  author = {Xiaoxia Wu and Zhewei Yao and Minjia Zhang and Conglong Li and Yuxiong He},
  title = {{XTC: Extreme Compression for Pre-trained Transformers Made Simple and Efficient}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=xNeAhc2CNAl},
}
Figure