Abstract
Along with the performance improvement in NLP domain, the sizes of transformer-based language models (TLM) are also dramatically increased. Some prior works intend to compress TLM models into more compact forms, but do not fully consider the hardware characters may not support the efficient execution for these forms, leading to the deployment of TLM on hardware with noticeable acceleration is still challenging. This paper thoroughly designs a compression scheme named GPUSQ-TLM to maximally utilize the GPU-friendly 2:4 fine-grained structured sparsity and quantization characters. Especially, a dense TLM model is first pruned to meet the GPU’s acceleration constraint of sparse patterns with FP16 type, then it is further quantized into a fixed-point one by quantization-aware training, to provide an extra speedup for integer tensors on GPU. A mixed-strategy knowledge distillation of labels, logits and feature maps is used for best accuracy compensation during pruning and quantization process. Experiment results show GPUSQ-TLM scheme achieves state-of-the-art compression on TLM model of various encoder and decoder blocks with negligible accuracy degradation on SQuAD, GLUE, CNN-DM & XSum and WikiText benchmarking tasks. Moreover, GPUSQ-TLM can boost actual deployment performance by up to 4.08-4.25x latency and 6.18-6.79x throughput on A100 GPU.
Introduction
The evolution of Natural Language Processing (NLP) has been defined by the rapid scaling of Transformer-based Language Models (TLMs). From BERT to GPT-3 and beyond, model sizes have grown from millions to hundreds of billions of parameters. While these massive models deliver state-of-the-art performance, their deployment in production environments is hindered by significant computational and memory requirements.
Existing compression techniques like pruning and quantization often suffer from an "efficiency gap." Irregular pruning patterns, while theoretically reducing model size, do not align with the dense matrix-math capabilities of modern GPUs, often resulting in negligible real-world speedups despite high theoretical pruning rates.
The GPUSQ-TLM Framework
To bridge this gap, researchers from Fudan University developed GPUSQ-TLM, a compression scheme designed specifically for NVIDIA’s Ampere architecture. The framework integrates three core components:
1. 2:4 Structured Sparsity
Unlike random pruning, GPUSQ-TLM enforces a "2:4" structured sparse pattern. In this pattern, for every four contiguous elements in a row, exactly two must be zero. This specific structure is natively supported by NVIDIA Ampere Tensor Cores, enabling the hardware to skip zero-value computations and effectively double the math throughput.
2. Quantization-Aware Training (QAT)
Following pruning, the model is further compressed from 16-bit floating point (FP16) to 8-bit integer (INT8) precision. To mitigate accuracy loss, the authors employ QAT, which simulates rounding errors during the training process, allowing the model to learn to be resilient to lower precision.
3. Mixed-Strategy Knowledge Distillation (KD)
To recover accuracy lost during compression, a dense "teacher" model guides the compressed "student" model. This distillation occurs at three levels: hard labels, soft logits (probability distributions), and internal feature maps. By forcing the student's internal representations to match the teacher's, the model retains the complex reasoning capabilities of the original architecture.
Experimental Results
The GPUSQ-TLM framework was evaluated across diverse benchmarks, including SQuAD 1.1, GLUE, WikiText-103, and text summarization tasks (CNN-DM, XSum).
- Hardware Acceleration: On an NVIDIA A100 GPU, the framework achieved a 4.08x to 4.25x reduction in latency and a 6.18x to 6.79x increase in throughput compared to dense baselines.
- Accuracy Preservation: The compression resulted in negligible accuracy drops. Notably, on SQuAD 1.1, the compressed BERT-base model showed a slight improvement (82.1 EM vs. 80.8 EM), likely due to the regularizing effect of the pruning process.
- Scalability: The method proved effective for various architectures, including encoder-only (BERT), decoder-only (GPT/OPT), and encoder-decoder (BART) models, even at scales exceeding 13 billion parameters.
Conclusion and Future Work
GPUSQ-TLM demonstrates that hardware-aware compression is essential for realizing the full potential of model pruning. By aligning software patterns with GPU architecture, significant speedups are achievable without sacrificing accuracy. While currently optimized for NVIDIA Ampere GPUs, future work may explore more aggressive sparsity patterns and extend these techniques to the "extreme" scale of trillion-parameter models.
Summary updated