NxMTransformer: Semi-Structured Sparsification for Natural Language Understanding via ADMM

Connor Holmes, Minjia Zhang, Yuxiong He, Bo Wu

TL;DR

NxMTransformer introduces an Alternating Direction Method of Multipliers (ADMM) framework to fine-tune dense, pretrained language models into hardware-friendly $N$:$M$ semi-structured sparse representations. By optimizing task loss and sparsity constraints alternately during downstream fine-tuning, it avoids the accuracy drop of one-shot pruning—achieving a 1.7-point GLUE score improvement over NVIDIA's standard ASP baseline while preserving hardware compatibility with Sparse Tensor Cores.

What Problem Does This Paper Address?

Large Transformer models (like BERT) deliver state-of-the-art natural language understanding, but their multi-million to multi-billion parameter sizes cause substantial latency and memory overheads during inference.

Modern accelerators (such as NVIDIA Ampere GPUs) introduced dedicated hardware for $N$:$M$ semi-structured sparsity (e.g., 2:4 sparsity, where exactly 2 out of every 4 consecutive values must be non-zero). This provides the hardware acceleration of structured pruning alongside the model expressiveness of unstructured pruning.

However, existing techniques to enforce $N$:$M$ sparsity fall short for pre-trained language models:

  1. One-shot magnitude pruning (e.g., NVIDIA's ASP) aggressively removes weights, degrading downstream transferability—especially on small, resource-constrained datasets.
  2. Training from scratch (e.g., SR-STE) requires re-running expensive pre-training purely for sparse checkpoints, abandoning compatibility with existing dense models.

Methods

NxMTransformer casts semi-structured sparsification as a constrained optimization problem and solves it during task-specific fine-tuning using ADMM:

$$\min_{\{W_i\}} f(\{W_i\}_{i=1}^L) + \sum_{i=1}^L g_i(Z_i) \quad \text{subject to } W_i = Z_i$$

Here, $f(\cdot)$ is the fine-tuning task objective, $Z_i$ is an auxiliary variable, and $g_i(\cdot)$ is a non-convex indicator penalty enforcing that $Z_i$ conforms to the $N$:$M$ pattern.

ADMM splits the optimization into alternating steps:

  • Sub-problem 1 (Performance-Promoting): Optimizes network weights $W_i$ using standard Adam on the task objective augmented with an $L_2$ regularization penalty pulling weights toward $(Z^k - U^k)$, where $U^k$ is the dual variable.
  • Sub-problem 2 (Sparsity-Promoting): Projects $(W^{k+1} + U^k)$ onto the $N$:$M$ constraint analytically in linear time by retaining the top $M$ absolute values in each contiguous group of $N$ weights.
  • Dual Variable Update: Updates $U^k \leftarrow U^{k-1} + W^k - Z^k$ to maintain dual feasibility.

Scope of Pruning: The method targets only the six dense linear layers in each Transformer block (Query, Key, Value, Attention Output, FFN1, and FFN2). The embedding layer and classification head are left dense, as embeddings cannot leverage Tensor Core sparse speedups and the task classifier is randomly initialized and sensitive to early pruning.

Datasets

The authors evaluate on the General Language Understanding Evaluation (GLUE) benchmark:

  • Large tasks: MNLI, SST-2, QNLI
  • Small / low-resource tasks: CoLA, STS-B, MRPC, RTE

Note: No new datasets were introduced.

Key Insights and Ideas

  • Continuous guidance beats abrupt pruning: Rather than zeroing weights in a single hard step, ADMM's alternating penalty smoothly pulls the dense model toward an $N$:$M$-compliant subspace while letting downstream representations adapt.
  • Mask stability protects pretrained knowledge: When tuning the ADMM penalty factor $\rho$, the authors observed that weights frequently swapping in and out of the sparse subnetwork mask suffer severe magnitude decay (~15% penalty per swap). High mask stability (>99% mask similarity across iterations) is critical to preventing destruction of pretrained features.
  • Small datasets benefit most from principled constraints: While data-rich tasks (like MNLI) have enough signal to recover from crude magnitude pruning, low-data tasks (like RTE and CoLA) drastically degrade unless optimization explicitly accounts for sparsity constraints.
  • Orthogonal to Knowledge Distillation (KD): Pruning 50% of weights across layers via 4:2 sparsity is less detrimental than dropping entire Transformer layers. Applying NxMTransformer on a distilled 6-layer model (DistilBERT) yields higher accuracy than standard DistilBERT at comparable parameter counts, demonstrating that intra-layer redundancy remains even after distillation.

Key Results

  • Overall GLUE Benchmark: On $\text{BERT}_{\text{base}}$ with 4:2 sparsity, NxMTransformer achieves an average GLUE score of 80.5, outperforming NVIDIA ASP (78.8) by 1.7 points and matching 99% of unstructured ADMM pruning (81.3) and 98.4% of the dense baseline (81.8).
  • Low-Resource Gains: On tasks with $<10\text{k}$ samples (CoLA, STS-B, MRPC, RTE), NxMTransformer beats ASP by an average of 2.9 points (and +1.9 points across 7 random seeds using median performance).
  • Combination with Distillation:
  • 12-layer 4:2 sparse $\text{BERT}_{\text{base}}$ (66.6M params) scores 80.5, beating the dense 6-layer DistilBERT (66.6M params) score of 78.5 by 2.0 points.
  • Applying NxMTransformer directly to DistilBERT yields a 45.3M parameter model scoring 76.6 (retaining 97.6% of DistilBERT's performance).

Limitations and Future Work

  • Not Lossless: While closing the gap with unstructured pruning, $N$:$M$ sparsity still incurs a modest accuracy drop compared to full dense models.
  • Unverified Outside NLP: The method was tested exclusively on text models (BERT, DistilBERT) and remains unverified on emerging Vision Transformers (ViTs).
  • Societal & Bias Considerations: The compressed model inherits upstream societal biases present in pretrained representations. Additionally, network pruning risks disproportionately degrading performance on minority or low-frequency inputs.

Summary updated

Abstract

Natural Language Processing (NLP) has recently achieved great success by using huge pre-trained Transformer networks. However, these models often contain hundreds of millions or even billions of parameters, bringing challenges to online deployment due to latency constraints. Recently, hardware manufacturers have introduced dedicated hardware for NxM sparsity to provide the flexibility of unstructured pruning with the runtime efficiency of structured approaches. NxM sparsity permits arbitrarily selecting M parameters to retain from a contiguous group of N in the dense representation. However, due to the extremely high complexity of pre-trained models, the standard sparse fine-tuning techniques often fail to generalize well on downstream tasks, which have limited data resources. To address such an issue in a principled manner, we introduce a new learning framework, called NxMTransformer, to induce NxM semi-structured sparsity on pretrained language models for natural language understanding to obtain better performance. In particular, we propose to formulate the NxM sparsity as a constrained optimization problem and use Alternating Direction Method of Multipliers (ADMM) to optimize the downstream tasks while taking the underlying hardware constraints into consideration. ADMM decomposes the NxM sparsification problem into two sub-problems that can be solved sequentially, generating sparsified Transformer networks that achieve high accuracy while being able to effectively execute on newly released hardware. We apply our approach to a wide range of NLP tasks, and our proposed method is able to achieve 1.7 points higher accuracy in GLUE score than current best practices. Moreover, we perform detailed analysis on our approach and shed light on how ADMM affects fine-tuning accuracy for downstream tasks. Finally, we illustrate how NxMTransformer achieves additional performance improvement with knowledge distillation based methods.
BibTeX
@inproceedings{holmes2021nxmtransformer,
  author = {Connor Holmes and Minjia Zhang and Yuxiong He and Bo Wu},
  title = {{NxMTransformer: Semi-Structured Sparsification for Natural Language Understanding via ADMM}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=bSgieZ8-be},
}
Figure