SFT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsity

Xinyu Yang, Jixuan Leng, Geyang Guo, Jiawei Zhao, Ryumei Nakada, Linjun Zhang, Huaxiu Yao, Beidi ChenNeurIPS 2024 · OpenReview

TL;DR

This paper introduces S2FT, a novel method for fine-tuning Large Language Models (LLMs) that achieves state-of-the-art performance, training efficiency, and scalable serving simultaneously. By strategically selecting and updating a sparse subset of parameters in a structured manner, S2FT significantly reduces memory usage and speeds up training and inference compared to existing methods like LoRA and full fine-tuning.

What Problem Does This Paper Address?

Fine-tuning Large Language Models (LLMs) for specific downstream tasks is crucial for their practical application. However, traditional full fine-tuning (FT) suffers from catastrophic forgetting, high computational costs, and impractical storage requirements for numerous fine-tuned models. Parameter-Efficient Fine-Tuning (PEFT) methods aim to address these issues by updating only a fraction of parameters. While existing PEFT methods excel in one or two aspects (e.g., high quality, efficient training, or scalable serving), none simultaneously achieve all three. This paper tackles this challenge by proposing S2FT, a method that aims to be high-quality, efficient, and scalable.

Methods

S2FT employs a "select sparsely and compute densely" strategy. It identifies coupled structures within LLMs, specifically in the Multi-Head Attention (MHA) and Feed-Forward Network (FFN) modules. The core idea is to:

  1. Sparse Selection: Select a small subset of attention heads in MHA and channels in FFN modules for fine-tuning.
  2. Co-permutation: Permute the weight matrices on both sides of these coupled structures to connect the selected subsets into a dense submatrix. This allows for dense gradient computation on the selected sparse components.
  3. Partial Back-propagation: Perform in-place gradient updates exclusively on these selected submatrices. This significantly reduces memory footprint and computation.

S2FT also proposes various strategies for selecting these sparse subsets, including random selection, selection based on weight magnitude, activation magnitude, gradient magnitude, and combinations thereof.

Datasets

The paper evaluates S2FT on a variety of datasets across different tasks:

  • Commonsense Reasoning: BoolQ, PIQA, SocialQA, HellaSwag, WinoGrande, ARC-challenge, ARC-easy, and OpenbookQA.
  • Arithmetic Reasoning: MultiArith, GSM8K, AddSub, AQUA, SingleEq, SVAMP, and MAWPS. The Math10K dataset, which combines training sets from GSM8K, MAWPS, and AQUA with augmented chain-of-thought steps, was used for training.
  • Instruction Following: Alpaca GPT-4 dataset (52k samples) for fine-tuning, evaluated on MT-Bench.

Key Insights and Ideas

  • Structured Sparsity is Key: The paper highlights that unstructured sparsity (like in traditional sparse fine-tuning) is inefficient on modern hardware. S2FT's structured approach, combined with dense computation, unlocks efficiency gains.
  • "Select Sparsely, Compute Densely": This core principle allows S2FT to achieve the benefits of sparsity (reduced parameters, memory) while leveraging the efficiency of dense matrix operations on GPUs.
  • Coupled Structures in LLMs: Identifying and exploiting the inherent dependencies between MHA and FFN modules is crucial for S2FT's effectiveness.
  • Generalization Advantage: S2FT demonstrates superior generalization capabilities compared to LoRA, particularly in out-of-distribution scenarios, suggesting that preserving more pre-trained knowledge is beneficial.
  • Scalable Serving Potential: The ability to decompose S2FT updates into adapters enables efficient fusion, switching, and parallelism, making it highly suitable for serving many fine-tuned models.

Key Results

  • Performance: S2FT achieves state-of-the-art performance, outperforming LoRA by 4.6% on commonsense reasoning and 1.3% on arithmetic reasoning. It also surpasses full FT by 11.5% on generalization tasks after instruction tuning.
  • Training Efficiency: S2FT saves 1.4-3.0x memory and achieves 1.5-2.7x speedup compared to full FT. It also offers a ~10% improvement in memory and latency over LoRA.
  • Inference Scalability: S2FT enables faster adapter switching and parallelism, with up to 22% faster inference than LoRA under similar memory constraints.
  • Adapter Fusion: S2FT shows better performance in adapter fusion compared to LoRA, especially when adapters are non-overlapped.

Limitations and Future Work

  • Architecture Extension: The authors suggest extending S2FT to other architectures like CNNs and RNNs.
  • Task Generalization: Exploring S2FT's applicability beyond language tasks, particularly in vision and multi-modal models, is a promising direction.
  • Selection Strategies: Further investigation into more sophisticated selection strategies could yield deeper insights.
  • Scaling: Scaling S2FT to even larger models requires further experimentation.
  • Serving System: Developing a practical serving system for S2FT is an important next step.

Summary updated

BibTeX
@inproceedings{yang2024sft,
  author = {Xinyu Yang and Jixuan Leng and Geyang Guo and Jiawei Zhao and Ryumei Nakada and Linjun Zhang and Huaxiu Yao and Beidi Chen},
  title = {{SFT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsity}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=mgJyPgZarH},
}
Figure