Less is More: Extreme Gradient Boost Rank-1 Adaption for Efficient Finetuning of LLMs

Yifei Zhang, Hao Zhu, Aiwei Liu, Han Yu, Piotr Koniusz, Irwin KingSubmission to ICLR 2025 · Withdrawn

TL;DR

XGBLoRA (eXtreme Gradient Boosting LoRA) adapts the classical machine learning principle of boosting to parameter-efficient fine-tuning: instead of training a single static low-rank adapter, it iteratively trains and merges a sequence of "weak" rank-1 adapters into the base model weights. This simple shift outperforms standard LoRA and full fine-tuning on diverse NLP benchmarks while slashing trainable parameters by over $10\times$ and allowing models like LLaMA3-8B to train comfortably on a single 24GB consumer GPU.

What Problem Does This Paper Address?

While Low-Rank Adaptation (LoRA) is the standard method for parameter-efficient fine-tuning (PEFT), it faces a persistent gap between theory and practice:

  • The theoretical dilemma: Recent expressive capacity analysis shows that to guarantee fitting any arbitrary target weight update, LoRA requires an adapter rank of $r \ge \text{embedding\_size}/2$.
  • The practical compromise: Real-world fine-tuning typically uses $r \in [8, 16]$ to keep memory footprints manageable, sacrificing representational capacity. Scaling up $r$ restores expressiveness but forfeits parameter and memory efficiency.

The authors ask: Can we retain the efficiency of ultra-low ranks without paying the performance penalty?

Methods

XGBLoRA resolves this trade-off by treating fine-tuning as an ensemble of iterative "weak learners":

  1. Iterative Boosting & Merging: Given a fixed training budget of $K$ total steps, training is split into $T$ iterations of $\kappa$ steps ($K = \kappa T$). In each iteration $t$, a small LoRA adapter $\Delta W^{(t)} = A^{(t)} B^{(t)\top}$ is trained on the model's residual errors. At the end of the iteration, the adapter is merged directly into the base weights ($W^{(t)} \leftarrow W^{(t-1)} + A^{(t)} B^{(t)\top}$) and reset.
  2. Rank-1 Weak Learners: Each booster is restricted to rank $r = 1$. Rather than attempting to learn the entire weight delta in one shot, individual rank-1 updates take fine-grained, incremental steps toward the optimum.
  3. Random Layer Subsampling: Inspired by feature subsampling in Gradient Boosted Decision Trees (GBDT), XGBLoRA randomly samples a subset of layers $L_s$ (e.g., 8 layers) to adapt during each boosting round. This enforces diversity across rounds and prevents individual adapters from overfitting.
  4. Theoretical Guarantees: The authors prove convergence and expressiveness error bounds showing that boosting iterations $T$ and rank $r$ can be traded off. An ensemble of rank-1 updates over $T$ iterations bounds the approximation error by $\mathcal{O}(1/r + 1/\sqrt{T})$, achieving the expressive power of much higher-rank adapters.
Base Model M_0 ──> [Train Adapter 1 (r=1, L_s layers) for κ steps] ──> Merge into M_1
                ──> [Train Adapter 2 (r=1, L_s layers) for κ steps] ──> Merge into M_2 ──> ...

Datasets

The authors evaluated XGBLoRA on established NLP benchmarks across multiple model architectures:

  • GLUE Benchmark: Evaluated on RoBERTa-base across 8 classification, similarity, and inference tasks (MNLI, SST-2, CoLA, QQP, QNLI, RTE, MRPC, STS-B).
  • Commonsense Reasoning: Evaluated across 8 sub-tasks (BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC-e, ARC-c, and OpenBookQA) using LLaMA-7B, LLaMA-13B, and LLaMA3-8B.
  • MMLU Benchmark: Evaluated across STEM, Social Sciences, Humanities, and Other subjects on LLaMA3-8B, Mistral-7B, and LLaMA2-13B after instruction tuning on the Alpaca GPT-4(en) dataset. (No new datasets were introduced.)

Key Insights and Ideas

  • The Weak Learner Analogue in PEFT: Just as decision trees perform best in boosting when restricted to shallow stumps, neural adapters benefit from intentional constraints. Ultra-low rank ($r=1$) combined with random layer dropping acts as an inductive bias and regularizer.
  • Trade Rank for Iterations at Zero Extra Step Cost: LoRA corresponds to $T = 1$ and $r \gg 1$. XGBLoRA flips this to $r = 1$ and $T \gg 1$ without increasing total optimization steps ($K = \kappa T$).
  • Substantial Hardware Democratization: Because backpropagation only maintains gradients and optimizer states for a rank-1 adapter across a fraction of layers, peak GPU memory drops significantly. LLaMA3-8B fine-tuning fits on a single NVIDIA RTX 4090 (24GB), whereas standard LoRA often requires a 40GB A100.

Key Results

  • GLUE Benchmark (RoBERTa-base): XGBLoRA achieves an average score of 87.85 with only 0.21‰ trainable parameters, outperforming standard LoRA (86.63, 2.65‰ params), DoRA (87.11), and full fine-tuning (86.87, 1000‰ params).
  • Commonsense Reasoning: On LLaMA3-8B, XGBLoRA reaches 83.0% average accuracy with 0.30‰ parameters, compared to LoRA's 80.5% with 7.00‰ parameters. Similar gains occur on LLaMA-7B (77.4% vs. 74.7%) and LLaMA-13B (81.8% vs. 80.5%).
  • MMLU Benchmark: Outperforms baselines across all base models, achieving 65.36% average accuracy on LLaMA3-8B with $r=1$ (vs. LoRA's 64.10% and full fine-tuning's 63.82%).
  • Efficiency: Shows lower latency per batch (~0.62s vs. 0.64s for LoRA) and lower peak VRAM consumption (~20GB vs. ~25GB).

Limitations and Future Work

  • Theoretical Assumptions: The formal proofs assume smoothness and strong convexity (simplifications standard in optimization proofs, but not reflective of the highly non-convex loss surfaces of deep transformers).
  • Tuning the Boosting Interval ($\kappa$): If $\kappa$ is set too small, individual boosters lack enough training steps to learn meaningful residual directions, leading to degraded performance.
  • Future Work: Extending the convergence theory to deeper non-linear dynamics, investigating adaptive selection of booster steps $\kappa$, and testing on multimodal foundation models.

Summary updated

Abstract

Fine-tuning Large Language Models (LLMs) has become a crucial technique for adapting pre-trained models to downstream tasks. However, the enormous size of LLMs poses significant challenges in terms of computational complexity and resource requirements. Low-Rank Adaptation (LoRA) has emerged as a promising solution. However, there exists a gap between the practical performance of low-rank adaptations and its theoretical optimum. In this work, we propose eXtreme Gradient Boosting LoRA (XGBLoRA), a novel framework that bridges this gap by leveraging the power of ensemble learning. Inspired by gradient boosting, XGBLoRA iteratively learns and merges a sequence of LoRA adaptations to refine model predictions. It achieves better performance than the standard LoRA, while enjoying the computational efficiency of rank-1 adaptations. We provide theoretical analysis to show the convergence and optimality of our approach, and conduct extensive experiments on a range of natural language processing tasks. The results demonstrate that XGBLoRA consistently outperforms standard LoRA and achieves performance comparable to full fine-tuning with significantly fewer trainable parameters. This work advances parameter-efficient fine-tuning for LLMs, and offers a promising solution for adapting LLMs to downstream tasks while optimizing performance and efficiency.
BibTeX
@misc{zhang2024less,
  author = {Yifei Zhang and Hao Zhu and Aiwei Liu and Han Yu and Piotr Koniusz and Irwin King},
  title = {{Less is More: Extreme Gradient Boost Rank-1 Adaption for Efficient Finetuning of LLMs}},
  year = {2024},
  eprint = {2410.19694},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.19694},
}
Figure