FineGates: LLMs Finetuning with Compression using Stochastic Gates

Jonathan Svirsky, Yehonathan Refael, Ofir LindenbaumarXiv 2024 · DOI · arXiv

Abstract

FineGates is a novel parameter-efficient finetuning (PEFT) method that integrates model compression directly into the adaptation process. By employing stochastic gates to introduce structured sparsity, FineGates enables the removal of 20–40% of base model parameters while maintaining competitive accuracy on downstream tasks. This approach addresses the inference bottleneck inherent in methods like LoRA, providing significant real-world speedups on standard hardware.

Introduction

Finetuning Large Language Models (LLMs) is a double-edged sword. Full finetuning is computationally expensive and prone to overfitting on small datasets. Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA (Low-Rank Adaptation) solve the training cost problem by adding a few trainable parameters while freezing the base model. However, LoRA has a major "inference bottleneck": because the base model remains frozen and intact, the model's size and inference speed at runtime remain exactly the same as the original. Existing compression techniques like pruning or quantization often require expensive post-training steps or specialized hardware. FineGates addresses this by asking: Can we adapt a model to a specific task and make it smaller/faster at the same time, in a single end-to-end process?

Methods

The core innovation of FineGates is the Stochastic Gate. Instead of just adding new weights, FineGates learns a vector of "gates" ($\omega$) that multiply the rows and columns of the base model's weight matrices.

  1. Stochastic Relaxation: Since binary gates (0 or 1) are non-differentiable, the authors use a Gaussian-based relaxation. They learn a continuous parameter ($\mu$) and add random noise during training, allowing gradients to flow through the "switches."
  2. Structured Sparsity: By applying these gates to entire rows or columns, the method achieves "structured" sparsity. This is crucial because modern hardware (CPUs/GPUs) can skip these zeroed-out blocks to save time, whereas "unstructured" (randomly scattered) zeros usually do not provide real-world speedups.
  3. The Objective Function: The training loss includes a standard task-specific term plus a sparsity regularization term. This forces the model to identify essential parts of the base model for the specific task.
Overview of FineGates training and inference workflow.

Experimental Setup

The authors evaluated FineGates using the GLUE benchmark on RoBERTa-base and RoBERTa-large architectures. They tested on small-scale sets (CoLA, STS-B, MRPC, RTE) and large-scale sets (MNLI, QQP, QNLI, SST-2). To simulate real-world "data-scarce" scenarios, large datasets were subsampled to 10,000 samples.

Key Results and Insights

FineGates successfully removed 20–40% of the base model's parameters in the attention layers. On the GLUE benchmark, FineGates performed comparably to (and sometimes better than) LoRA and full finetuning, despite having a significantly smaller footprint at inference time.

Performance comparison of FineGates against LoRA and full finetuning on GLUE tasks.
  • Sparsification as Adaptation: A key finding is that simply removing unimportant parts of a pre-trained model can be a form of adaptation itself.
  • End-to-End Efficiency: Unlike traditional pruning which requires a "train-prune-retrain" cycle, FineGates compresses the model during the initial finetuning.
  • Hardware-Friendly Speedups: Because the sparsity is structured, the authors demonstrated real-world wall-clock time reductions. A 40% reduction in parameters led to a nearly 30% decrease in inference latency on standard hardware like an Intel i9 CPU or an NVIDIA RTX 3080.

Limitations and Future Work

While the principles should apply to decoder-only models (like Llama or GPT), FineGates has primarily been tested on RoBERTa. Future research will explore extending the framework to prune entire attention heads and investigating its utility in multi-task finetuning scenarios, where different gates might activate different model components for different tasks.

Summary updated

Abstract

Large Language Models (LLMs), with billions of parameters, present significant challenges for full finetuning due to the high computational demands, memory requirements, and impracticality of many real-world applications. When faced with limited computational resources or small datasets, updating all model parameters can often result in overfitting. To address this, lightweight finetuning techniques have been proposed, like learning low-rank adapter layers. These methods aim to train only a few additional parameters combined with the base model, which remains frozen, reducing resource usage and mitigating overfitting risks. In this work, we propose an adaptor model based on stochastic gates that simultaneously sparsify the frozen base model with task-specific adaptation. Our method comes with a small number of trainable parameters and allows us to speed up the base model inference with competitive accuracy. We evaluate it in additional variants by equipping it with additional low-rank parameters and comparing it to several recent baselines. Our results show that the proposed method improves the finetuned model accuracy comparatively to the several baselines and allows the removal of up to 20-40\% without significant accuracy loss.
BibTeX
@misc{svirsky2024finegates,
  author = {Jonathan Svirsky and Yehonathan Refael and Ofir Lindenbaum},
  title = {{FineGates: LLMs Finetuning with Compression using Stochastic Gates}},
  year = {2024},
  eprint = {2412.12951},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.12951},
}
Figure