Compressing Pre-trained Transformers via Low-Bit NxM Sparsity for Natural Language Understanding

Connor Holmes, Minjia Zhang, Yuxiong He, Bo WuarXiv 2022 · DOI · arXiv

Abstract

Large-scale Transformers have revolutionized Natural Language Understanding (NLU) but present significant deployment challenges due to their size and computational demands. This paper introduces NxMiFormer, a compression framework that jointly optimizes N:M semi-structured sparsity and low-bit quantization. By leveraging an efficient heterogeneous search algorithm to identify layer-wise sensitivity, NxMiFormer achieves up to 93.75% compression of BERT encoders while maintaining over 98% of baseline accuracy, resulting in a 2.5x inference speedup on modern hardware.

Introduction

While models like BERT and RoBERTa have set new benchmarks in NLU, their massive parameter counts make them difficult to deploy on edge devices or in latency-sensitive environments. Previous attempts to mitigate this through pruning or quantization often treat these techniques in isolation. Furthermore, standard unstructured pruning often fails to translate into real-world speedups on modern GPUs.

NxMiFormer addresses these issues by jointly optimizing for both sparsity and quantization. Specifically, it targets the Sparse Tensor Cores in NVIDIA’s Ampere architecture, which are optimized for "N:M" sparsity (e.g., 2:4 sparsity, where exactly 2 out of every 4 weights are zero).

Methodology

The NxMiFormer framework integrates two primary optimization techniques:

  1. ADMM-based Sparsification: The Alternating Direction Method of Multipliers (ADMM) is used to enforce N:M hardware constraints by breaking the complex sparsity problem into manageable sub-problems.
  2. STE-based Quantization-Aware Training (QAT): A Straight-Through Estimator (STE) allows the model to learn optimal weights for low-precision formats (like INT8 or INT4) during the fine-tuning process.
Ablation study comparing different optimization methods (ADMM vs. STE-QAT) and bit-widths across GLUE tasks.

A critical component of the framework is the Heterogeneous Search Algorithm. Recognizing that Transformer layers exhibit varying sensitivity to compression—for instance, the second feed-forward (FFN) layer is typically resilient, while Query and Value matrices are highly sensitive—the authors developed a fast heuristic. This "Immediate Loss" proxy measures the validation loss immediately after applying constraints, accurately predicting post-training performance without the need for full retraining.

Experimental Results

The framework was evaluated on the GLUE benchmark and SQuAD 1.1. Results demonstrate that NxMiFormer significantly outperforms single-method approaches like Q-BERT (quantization only) or NxMTransformer (sparsity only).

Comparison of NxMiFormer against baseline and state-of-the-art compression methods on GLUE tasks.

Key performance metrics include:

  • Compression Efficiency: NxMiFormer-Search compressed the BERT encoder stack by 93.75% while retaining 98.2% of the baseline accuracy.
  • Hardware Acceleration: End-to-end testing on an NVIDIA A100 GPU via TensorRT showed up to a 2.5x speedup for BERT-Large.
  • Search Speed: The heuristic search is highly efficient, identifying optimal configurations for tasks like CoLA in under an hour on a single V100 GPU.

Discussion and Limitations

The authors found that sparsity and quantization act orthogonally, allowing for effective joint optimization. However, several avenues for future work remain. The current study focused on NLU tasks; extending this to generative (GPT-style) or Vision Transformers is a logical next step. Additionally, while theoretical 4-bit efficiency is high, current production runtimes often up-cast to 8-bit, limiting realized gains. Finally, the authors highlight the importance of monitoring "fairness-aware" compression, as aggressive pruning can sometimes amplify underlying social biases in the data.

Summary updated

Abstract

In recent years, large pre-trained Transformer networks have demonstrated dramatic improvements in many natural language understanding tasks. However, the huge size of these models brings significant challenges to their fine-tuning and online deployment due to latency and cost constraints. New hardware supporting both N:M semi-structured sparsity and low-precision integer computation is a promising solution to boost DNN model serving efficiency. However, there have been very few studies that systematically investigate to what extent pre-trained Transformer networks benefit from the combination of these techniques, as well as how to best compress each component of the Transformer. We propose a flexible compression framework NxMiFormer that performs simultaneous sparsification and quantization using ADMM and STE-based QAT. Furthermore, we present and inexpensive, heuristic-driven search algorithm that identifies promising heterogeneous compression configurations that meet a compression ratio constraint. When evaluated across the GLUE suite of NLU benchmarks, our approach can achieve up to 93% compression of the encoders of a BERT model while retaining 98.2% of the original model accuracy and taking full advantage of the hardware's capabilities. Heterogeneous configurations found the by the search heuristic maintain 99.5% of the baseline accuracy while still compressing the model by 87.5%.
BibTeX
@misc{holmes2022compressing,
  author = {Connor Holmes and Minjia Zhang and Yuxiong He and Bo Wu},
  title = {{Compressing Pre-trained Transformers via Low-Bit NxM Sparsity for Natural Language Understanding}},
  year = {2022},
  eprint = {2206.15014},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2206.15014},
}
Figure