Less is More: Extreme Gradient Boost Rank-1 Adaption for Efficient Finetuning of LLMs

Yifei Zhang, Hao Zhu, Aiwei Liu, Han Yu, Piotr Koniusz, Irwin King

Abstract

XGBLoRA (eXtreme Gradient Boosting LoRA) reimagines Large Language Model (LLM) fine-tuning by treating low-rank adapters as "weak learners" in a gradient boosting ensemble. By iteratively merging rank-1 adaptations into the base model, XGBLoRA achieves superior performance compared to standard LoRA and even full fine-tuning, while utilizing up to 10x fewer trainable parameters and significantly reducing GPU memory requirements.

Introduction: Bridging the Efficiency-Performance Gap

Fine-tuning Large Language Models (LLMs) remains a resource-intensive challenge. Low-Rank Adaptation (LoRA) has emerged as the standard for Parameter-Efficient Fine-Tuning (PEFT), yet it faces a significant "theoretical-practical gap." While theory suggests that matching a target weight matrix requires a rank ($r$) proportional to the embedding dimension, practical constraints often limit $r$ to small values (e.g., 8 or 16) to conserve memory. This limitation often results in a performance penalty. XGBLoRA addresses this by asking: Is there an efficient way to bridge this performance gap while maintaining the extreme low-rank structure and low complexity of LoRA?

Methodology: The XGBLoRA Framework

Inspired by Gradient Boosting Decision Trees (GBDT), XGBLoRA shifts the paradigm from training a single "strong" adapter to building an ensemble of many "weak" adapters. The framework operates through an iterative process:

  1. Learn Rank-1 Adaptations: In each boosting iteration, the model learns a set of LoRA matrices with a rank of exactly 1.
  2. Random Layer Selection: To maintain the "weak learner" property and prevent overfitting, only a random subset of layers is updated per iteration.
  3. Iterative Merging: After training a booster for a fixed number of steps ($\kappa$), it is merged directly into the base model's weights ($W \leftarrow W + AB$). Subsequent boosters then target the "residual error" of the updated base model.
XGBLoRA training process: iterative rank-1 adaptation and merging.

This approach allows the model to progressively capture task complexity without ever requiring high-rank updates, significantly lowering the memory footprint at any single point in time.

Experimental Evaluation

The researchers evaluated XGBLoRA across diverse benchmarks, including the GLUE benchmark for natural language understanding (using RoBERTa-base), and various commonsense reasoning tasks (using LLaMA-7B, 13B, and LLaMA3-8B).

Key Results

XGBLoRA demonstrates a remarkable balance between performance and efficiency. On the GLUE benchmark, it achieved an average score of 87.85, surpassing both standard LoRA (86.63) and Full Fine-Tuning (86.87).

Performance vs. parameter efficiency on GLUE benchmark.

In LLaMA3-8B tests, XGBLoRA used only 0.30‰ of the parameters compared to LoRA's 7.00‰, yet achieved higher average scores. Furthermore, the efficiency of rank-1 updates enables fine-tuning an 8B model on consumer-grade hardware like the NVIDIA RTX 4090, which typically lacks the memory for standard LoRA at that scale.

Key Insights: Why "Weakness" Works

The success of XGBLoRA stems from two core principles:

  • Regularization through Simplicity: By limiting rank to 1 and using random layer selection, XGBLoRA acts as a natural regularizer, making it resilient to noise and overfitting.
  • Iteration as a Substitute for Rank: The authors provide formal proofs showing that increasing boosting iterations ($T$) can mathematically compensate for lower rank ($r$). Many simple steps effectively reach the same optimization goal as fewer complex ones.

Conclusion

While XGBLoRA shows exceptional empirical results, the current theoretical analysis relies on linear model assumptions. Future work aims to extend this theory to non-linear systems. Nonetheless, XGBLoRA represents a significant step toward making high-performance LLM adaptation accessible on commodity hardware.

Summary updated

Abstract

Fine-tuning Large Language Models (LLMs) has become a crucial technique for adapting pre-trained models to downstream tasks. However, the enormous size of LLMs poses significant challenges in terms of computational complexity and resource requirements. Low-Rank Adaptation (LoRA) has emerged as a promising solution. However, there exists a gap between the practical performance of low-rank adaptations and its theoretical optimum. In this work, we propose eXtreme Gradient Boosting LoRA (XGBLoRA), a novel framework that bridges this gap by leveraging the power of ensemble learning. Inspired by gradient boosting, XGBLoRA iteratively learns and merges a sequence of LoRA adaptations to refine model predictions. It achieves better performance than the standard LoRA, while enjoying the computational efficiency of rank-1 adaptations. We provide theoretical analysis to show the convergence and optimality of our approach, and conduct extensive experiments on a range of natural language processing tasks. The results demonstrate that XGBLoRA consistently outperforms standard LoRA and achieves performance comparable to full fine-tuning with significantly fewer trainable parameters. This work advances parameter-efficient fine-tuning for LLMs, and offers a promising solution for adapting LLMs to downstream tasks while optimizing performance and efficiency.
BibTeX
@misc{zhang2024less,
  author = {Yifei Zhang and Hao Zhu and Aiwei Liu and Han Yu and Piotr Koniusz and Irwin King},
  title = {{Less is More: Extreme Gradient Boost Rank-1 Adaption for Efficient Finetuning of LLMs}},
  year = {2024},
  eprint = {2410.19694},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.19694},
}
Figure