Shears: Unstructured Sparsity with Neural Low-rank Adapter Search

J. Pablo Muñoz, Jinjie Yuan, Nilesh Jain

Abstract

Recently, several approaches successfully demonstrated that weight-sharing Neural Architecture Search (NAS) can effectively explore a search space of elastic low-rank adapters (LoRA), allowing the parameter-efficient fine-tuning (PEFT) and compression of large language models. In this paper, we introduce a novel approach called Shears, demonstrating how the integration of cost-effective sparsity and a proposed Neural Low-rank adapter Search (NLS) algorithm can further improve the efficiency of PEFT approaches. Results demonstrate the benefits of Shears compared to other methods, reaching high sparsity levels while improving or with little drop in accuracy, utilizing a single GPU for a pair of hours.

Introduction

As Large Language Models (LLMs) scale to hundreds of billions of parameters, deploying them on consumer hardware or edge devices becomes increasingly prohibitive. Two primary strategies have emerged to mitigate this: Parameter-Efficient Fine-Tuning (PEFT), such as LoRA, which minimizes trainable parameters, and Model Compression, which reduces total parameter count through pruning.

However, effectively combining these strategies is challenging. Pruning often leads to performance degradation that standard fine-tuning may fail to recover. Moreover, many existing sparse fine-tuning methods are computationally expensive. Shears addresses these issues by combining cost-effective unstructured pruning with "elastic" low-rank adapters to create lean, task-specific models.

Conceptual overview of Shears: combining pruned, frozen pre-trained weights with an elastic adapter.

Methodology

The Shears framework operates through a three-stage pipeline designed for efficiency and performance retention:

  1. Unstructured Sparsification: The process begins with the Wanda algorithm, a "zeroth-order" approach that prunes weights based on magnitude and input activation norms. This step is highly efficient, processing a 7B parameter model in under five minutes on a single GPU.
  2. Super-Adapter Training: Instead of fixed-rank LoRA adapters, Shears employs elastic adapters. A "super-adapter" network is trained where different ranks (widths) can be sampled. This weight-sharing mechanism allows the model to learn which layers require higher capacity to compensate for pruning-induced information loss.
  3. Sub-Adapter Search (NLS): Finally, Neural Low-rank adapter Search (NLS) is used to identify the optimal adapter configuration. Using either a heuristic or a cost-effective hill-climbing algorithm, Shears determines the specific rank for each layer that maximizes task accuracy.
The three-stage Shears workflow: Sparsification, Super-Adapter Training, and Sub-Adapter Search.

Experimental Evaluation

The authors evaluated Shears using LLaMA (7B, 13B) and MPT-7B models across several benchmarks:

  • Math Reasoning: GSM8K, AQuA, MAWPS, and SVAMP.
  • Commonsense Reasoning: A suite of eight datasets including BoolQ, PIQA, ARC, and OBQA.
  • Training Data: High-quality unified datasets (10k to 170k samples) generated via GPT-3.5 were used for fine-tuning.

Key Results and Insights

  • Performance Retention: On math reasoning tasks, LLaMA-7B with 40% sparsity achieved an average accuracy of 46.9%, matching the performance of a standard, dense LoRA model. For LLaMA-13B, Shears reached 50% sparsity with 50.9% accuracy (vs. 51.1% for dense LoRA), effectively halving the non-zero parameters.
  • Sparsity as a Regularizer: At 40% sparsity, Shears often outperformed dense PEFT baselines, suggesting that the combination of pruning and targeted adapter search can act as an effective regularizer.
  • Efficiency: Unlike methods requiring full-model retraining (e.g., SparseFT), Shears only updates adapters, significantly reducing compute requirements. In head-to-head tests on GSM8K, Shears consistently outperformed SparseFT across all sparsity levels while being cheaper to train.
  • The Role of Elasticity: Searching for different ranks across layers is crucial. NLS allows the model to allocate trainable parameters where they are most needed to recover performance from pruning.

Conclusion and Future Work

Shears demonstrates that high levels of unstructured sparsity can be achieved in LLMs with minimal accuracy loss using a single GPU. While unstructured sparsity offers theoretical compression, future work will explore structured pruning to realize wall-clock speedups on standard hardware. Additionally, the authors suggest that combining Shears with quantization (e.g., 4-bit weights) could push the boundaries of model compression even further.

Summary updated

Abstract

Recently, several approaches successfully demonstrated that weight-sharing Neural Architecture Search (NAS) can effectively explore a search space of elastic low-rank adapters (LoRA), allowing the parameter-efficient fine-tuning (PEFT) and compression of large language models. In this paper, we introduce a novel approach called Shears, demonstrating how the integration of cost-effective sparsity and a proposed Neural Low-rank adapter Search (NLS) algorithm can further improve the efficiency of PEFT approaches. Results demonstrate the benefits of Shears compared to other methods, reaching high sparsity levels while improving or with little drop in accuracy, utilizing a single GPU for a pair of hours.
BibTeX
@inproceedings{munoz2024shears,
  author = {J. Pablo Muñoz and Jinjie Yuan and Nilesh Jain},
  title = {{Shears: Unstructured Sparsity with Neural Low-rank Adapter Search}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=irVgunvRfi},
}
Figure