NxMTransformer: Semi-Structured Sparsification for Natural Language Understanding via ADMM

Connor Holmes, Minjia Zhang, Yuxiong He, Bo Wu

Introduction: The Efficiency Challenge in NLP

Large-scale Transformer models like BERT have set new benchmarks in Natural Language Understanding (NLU). However, their massive parameter counts—often in the hundreds of millions—present significant deployment hurdles regarding latency and computational cost. To mitigate this, researchers employ sparsification techniques to remove redundant weights, but they often face a trade-off between hardware efficiency and model accuracy.

The Sparsity Dilemma

Traditionally, sparsification follows two paths:

  1. Unstructured Sparsity: Random weights are removed. While this preserves accuracy, it results in irregular matrices that modern hardware cannot process efficiently.
  2. Structured Sparsity: Entire rows or columns are removed. This is hardware-friendly but often degrades accuracy significantly by deleting critical information.

Recently, hardware manufacturers introduced NxM semi-structured sparsity (specifically 4:2 sparsity), which requires keeping exactly 2 out of every 4 contiguous weights. This format offers a middle ground, but existing methods like NVIDIA’s Automatic Sparsity (ASP) often struggle to maintain performance when fine-tuning on small downstream datasets.

Proposed Method: NxMTransformer

The authors introduce NxMTransformer, a framework that reformulates sparsification as a constrained optimization problem using the Alternating Direction Method of Multipliers (ADMM).

The NxMTransformer framework uses ADMM to transform a dense pretrained model into a hardware-compatible sparse representation.

Instead of simple magnitude pruning, ADMM decomposes the problem into two iterative sub-problems:

  • Performance Optimization: The model is fine-tuned on the target task while a dynamic penalty is applied to weights violating the NxM constraint.
  • Sparsity Projection: Weights are projected onto the NxM constraint by retaining the $M$ largest values in each block.

This iterative approach allows the model to gradually adapt to the sparse structure, preserving the knowledge embedded during pre-training more effectively than "one-shot" pruning methods.

Key Results and Insights

The framework was evaluated on the GLUE (General Language Understanding Evaluation) benchmark, covering tasks ranging from the large-scale MNLI to the small-scale RTE.

Performance comparison of NxMTransformer against baseline and ASP across GLUE tasks.

Performance Highlights

  • Superior Accuracy on Small Data: NxMTransformer excels when training data is limited. On small tasks like CoLA and RTE, it outperformed the ASP method by an average of 2.9 points.
  • Overall Benchmark Success: The method achieved a GLUE score 1.7 points higher than the ASP baseline, retaining 99% of the accuracy of unstructured sparsity while enabling hardware speedups of 1.3x to 1.6x on NVIDIA A100 GPUs.
  • Mask Stability: The researchers found that ADMM stabilizes the "mask" (the pattern of retained weights). Higher similarity between masks across training iterations correlates strongly with better final accuracy, suggesting that preventing weight "jitter" is key to preserving pre-trained knowledge.

Conclusion

NxMTransformer demonstrates that principled optimization via ADMM can bridge the gap between hardware-friendly structured sparsity and accuracy-preserving unstructured methods. By baking hardware constraints directly into the fine-tuning process, it enables the deployment of high-performance Transformers on modern accelerators without the typical accuracy trade-offs associated with specialized downstream tasks.

Summary updated

BibTeX
@inproceedings{holmes2021nxmtransformer,
  author = {Connor Holmes and Minjia Zhang and Yuxiong He and Bo Wu},
  title = {{NxMTransformer: Semi-Structured Sparsification for Natural Language Understanding via ADMM}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=bSgieZ8-be},
}
Figure