Train Less, Infer Faster: Efficient Model Finetuning and Compression via Structured Sparsity

Jonathan Svirsky, Yehonathan Refael, Ofir Lindenbaum

TL;DR

This paper introduces "FineGates," a novel method for efficiently adapting large language models (LLMs) to new tasks. By learning to selectively prune (sparsify) rows and columns of the LLM's weight matrices using "stochastic gates," FineGates achieves significant reductions in model size and inference time without sacrificing accuracy, outperforming existing methods like LoRA.

What Problem Does This Paper Address?

Training and fine-tuning massive foundation language models (LLMs) with billions of parameters is computationally expensive, requires substantial memory, and can easily lead to overfitting, especially with limited task-specific data. While methods like Low-Rank Adaptation (LoRA) reduce the number of trainable parameters, they don't decrease the inference latency or memory footprint of the underlying large model, as the full model still needs to be processed. The challenge is to find a way to adapt LLMs efficiently, making them smaller and faster for inference, without compromising their performance.

Methods

The core innovation of this paper is FineGates, a structured sparsification technique. Instead of adding new trainable parameters (like LoRA adapters), FineGates learns binary "stochastic gates" that selectively zero out entire rows and columns of the pre-trained LLM's weight matrices. These gates are trained end-to-end, allowing the model to identify and retain only the most crucial parameters for a specific task. This process effectively prunes the model during fine-tuning, leading to a smaller, more efficient model for inference. The method uses a reparameterization trick to approximate binary gates with continuous variables during training and incorporates a sparsity regularization term to control the degree of pruning.

Datasets

The authors evaluated their method on the GLUE benchmark, a standard suite of natural language understanding tasks. They used both the full GLUE datasets and a subset with up to 10,000 samples to simulate scenarios with limited data. They also conducted experiments on the C4 dataset for pre-training and evaluated on the MRPC dataset for inference speedup measurements.

Key Insights and Ideas

  • Sparsity as a direct adaptation mechanism: The paper demonstrates that structured sparsity, achieved through learned gates, can directly facilitate task adaptation without requiring weight tuning or adding new parameters. This is a departure from many existing PEFT (Parameter-Efficient Fine-Tuning) methods.
  • Inference-time efficiency is achievable: Unlike LoRA, FineGates directly reduces the computational graph and memory footprint of the model, leading to tangible inference speedups.
  • Simpler optimization landscape: Theoretical analysis suggests that FineGates offers a more stable and better-conditioned optimization landscape compared to LoRA, potentially leading to more reliable convergence.
  • Later layers are more amenable to pruning: Empirical observations show that learned gates tend to become sparse earlier in training for later layers of the Transformer, aligning with prior findings on pruning.

Key Results

  • FineGates can remove 20-40% of model parameters without significant accuracy loss across various tasks and backbones (RoBERTa-Base, RoBERTa-Large, Llama3.2-1B).
  • On the MRPC task, FineGates achieved up to 25% inference time reduction with only a 4% accuracy loss.
  • FineGates outperforms recent finetuning baselines in efficiency and performance, often achieving comparable or better accuracy than LoRA with significantly fewer trainable parameters.
  • The method demonstrated effectiveness in pre-training scenarios, enabling faster convergence and improved inference efficiency.

Limitations and Future Work

  • Limited to GLUE benchmark: The authors acknowledge that their experiments are primarily restricted to the GLUE benchmark due to computational constraints. They suggest that evaluating on larger models and more diverse tasks would be valuable.
  • CPU-centric optimization: While the method achieves structured sparsity suitable for CPU inference, further work could explore optimizations for GPU architectures.
  • Future directions: The paper suggests exploring FineGates for other applications like model compression, quantization, and further investigation into its theoretical properties.

Summary updated

Abstract

Fully finetuning foundation language models (LMs) with billions of parameters is often impractical due to high computational costs, memory requirements, and the risk of overfitting. Although methods like low-rank adapters help address these challenges by adding small trainable modules to the frozen LM, they also increase memory usage and do not reduce inference latency. We uncover an intriguing phenomenon: sparsifying specific model rows and columns enables efficient task adaptation without requiring weight tuning. We propose a scheme for effective finetuning via sparsification using training stochastic gates, which requires minimal trainable parameters, reduces inference time, and removes 20--40\% of model parameters without significant accuracy loss. Empirical results show it outperforms recent finetuning baselines in efficiency and performance. Additionally, we provide theoretical guarantees for the convergence of this stochastic gating process, and show that our method admits a simpler and better-conditioned optimization landscape compared to LoRA. Our results highlight sparsity as a compelling mechanism for task-specific adaptation in LMs.
BibTeX
@inproceedings{svirsky2026train,
  author = {Jonathan Svirsky and Yehonathan Refael and Ofir Lindenbaum},
  title = {{Train Less, Infer Faster: Efficient Model Finetuning and Compression via Structured Sparsity}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2026},
  url = {https://openreview.net/forum?id=jU4ERfrjpH},
}
Figure