Shears: Unstructured Sparsity with Neural Low-rank Adapter Search

J. Pablo Muñoz, Jinjie Yuan, Nilesh Jain

Abstract

Recently, several approaches successfully demonstrated that weight-sharing Neural Architecture Search (NAS) can effectively explore a search space of elastic low-rank adapters (LoRA), allowing the parameter-efficient fine-tuning (PEFT) and compression of large language models. In this paper, we introduce a novel approach called Shears, demonstrating how the integration of cost-effective sparsity and a proposed Neural Low-rank adapter Search (NLS) algorithm can further improve the efficiency of PEFT approaches. Results demonstrate the benefits of Shears compared to other methods, reaching high sparsity levels while improving or with little drop in accuracy, utilizing a single GPU for a pair of hours.

Introduction

As Large Language Models (LLMs) scale to hundreds of billions of parameters, deploying them on consumer-grade hardware becomes increasingly prohibitive. To address this, researchers typically employ two distinct strategies: Model Compression (such as pruning or sparsification) to reduce the physical size of the model, and Parameter-Efficient Fine-Tuning (PEFT) (such as LoRA) to adapt models to specific tasks without retraining the entire weight set.

However, combining these strategies effectively is non-trivial. Standard pruning often leads to significant accuracy degradation, and traditional fine-tuning does not reduce the memory footprint of the original "frozen" weights. Existing "sparse fine-tuning" methods often demand massive computational resources or complex knowledge distillation. Shears is introduced as a practical solution that delivers high sparsity and high performance without the massive compute bill, achieving up to 50% sparsity with minimal accuracy loss in just a few hours on a single GPU.

The Shears Framework

The Shears workflow is a three-stage pipeline designed to maximize efficiency and performance synergy between pruning and adaptation.

The three-stage Shears framework: Unstructured Sparsification, Super-Adapter Training, and Sub-Adapter Search.

1. Unstructured Sparsification

Instead of computationally expensive iterative pruning, Shears utilizes the Wanda algorithm. This is a "zeroth-order" method that requires no training. It calculates weight importance based on the product of weight magnitude and input activations. This allows the framework to zero out 40-50% of a 7B parameter model in less than five minutes.

2. Super-Adapter Training

The framework introduces "elastic" LoRA adapters. Unlike standard LoRA, which uses a fixed rank (e.g., 8 or 16) for every layer, a "super-adapter" is trained to support multiple ranks simultaneously. This enables the model to learn which specific layers require more capacity to compensate for the information lost during sparsification.

3. Neural Low-rank adapter Search (NLS)

Finally, a specialized search algorithm identifies the optimal rank for each adapter across the model. To avoid the computational explosion of searching every possible combination, Shears employs a heuristic starting point followed by a cost-effective hill-climbing algorithm to fine-tune the configuration. This ensures that the final "sub-adapter" is optimized for the specific task and sparsity level.

Experimental Evaluation

The researchers evaluated Shears across a wide variety of benchmarks and models:

  • Models: LLaMA (7B and 13B) and MPT-7B.
  • Math Reasoning: GSM8K, AQuA, MAWPS, and SVAMP.
  • Commonsense Reasoning: A suite of eight datasets including BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC (Easy and Challenge), and OBQA.

Key Results

  • Accuracy Retention: On LLaMA-13B for math tasks, Shears with 40% sparsity actually outperformed standard, non-sparse LoRA (52.0% vs 51.1% average accuracy). Even at 50% sparsity, the model remained highly competitive.
  • Parameter Reduction: For a LLaMA-7B model, Shears successfully reduced the non-zero parameter count from 6.7B to 3.5B while maintaining 45.3% accuracy on math tasks (compared to 46.9% for a full-dense LoRA model).
  • Efficiency: Unlike other state-of-the-art sparse fine-tuning methods that require full-model distillation and FP32 precision, Shears works with FP16 and only updates a tiny fraction of parameters. This allows the entire process to run on a single GPU in approximately two hours.

Discussion and Insights

The primary insight of Shears is the synergy between sparsification and elastic adaptation. By sparsifying the base model before adding adapters, the adapters can effectively "compensate" for the missing information. The search phase (NLS) revealed that not all layers are created equal; by allowing different layers to have different LoRA ranks, Shears maintains higher accuracy at high sparsity levels compared to uniform rank approaches.

Conclusion and Future Work

While Shears significantly reduces the number of non-zero parameters, realizing actual inference speedups currently requires specialized hardware or software kernels capable of skipping zero-value computations. Future research directions include combining this approach with structured sparsity (pruning entire blocks or neurons) to achieve direct hardware-level speedups without specialized kernels. Additionally, while the hill-climbing search is efficient, more refined sub-adapter configurations could potentially be discovered through more intensive evolutionary search strategies for users with larger compute budgets.

Summary updated

Abstract

Recently, several approaches successfully demonstrated that weight-sharing Neural Architecture Search (NAS) can effectively explore a search space of elastic low-rank adapters (LoRA), allowing the parameter-efficient fine-tuning (PEFT) and compression of large language models. In this paper, we introduce a novel approach called Shears, demonstrating how the integration of cost-effective sparsity and a proposed Neural Low-rank adapter Search (NLS) algorithm can further improve the efficiency of PEFT approaches. Results demonstrate the benefits of Shears compared to other methods, reaching high sparsity levels while improving or with little drop in accuracy, utilizing a single GPU for a pair of hours.
BibTeX
@inproceedings{munoz2024shears,
  author = {J. Pablo Muñoz and Jinjie Yuan and Nilesh Jain},
  title = {{Shears: Unstructured Sparsity with Neural Low-rank Adapter Search}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=irVgunvRfi},
}
Figure