LoRASC: Expressive and Generalizable Low-rank Adaptation for Large Models via Slow Cascaded Learning

Siwei Li, Yifan Yang, Yifei Shen, Fangyun Wei, Zongqing Lu, Lili Qiu, Yuqing Yang

Abstract

LoRASC (Low-Rank Adaptation via Slow Cascaded Learning) is a novel fine-tuning framework designed to enhance the expressiveness and generalization of Low-Rank Adaptation (LoRA) without introducing inference overhead. By iteratively training and merging cascaded low-rank experts through a slow-fast update mechanism, LoRASC achieves state-of-the-art performance across diverse language and vision benchmarks while effectively mitigating common overfitting issues associated with standard Parameter-Efficient Fine-Tuning (PEFT) methods.

Introduction

Fine-tuning massive foundation models is computationally demanding, leading to the widespread adoption of PEFT methods like LoRA. LoRA reduces trainable parameters by adding small, low-rank matrices to model layers. However, LoRA faces three primary bottlenecks: limited expressiveness due to low-rank constraints, a tendency to overfit in low-data regimes, and high sensitivity to hyperparameters like rank and learning rate. LoRASC addresses these limitations by rethinking the scheduling and update mechanisms of low-rank modules during training.

Methodology

LoRASC introduces three synergistic components to improve adaptation:

  1. Cascading LoRA Learning: Instead of a single static LoRA module, LoRASC trains a new "expert" module every epoch. Each expert is merged into the backbone weights at the epoch's end, and a new one is initialized to learn from the remaining residuals. This allows the model to accumulate a higher total rank over time without increasing the memory footprint of any single step.
  2. Slow-Fast Update Mechanism: Inspired by Stochastic Weight Averaging (SWA), LoRASC maintains a "fast" expert that learns directly from data and a "slow" expert, which is an Exponential Moving Average (EMA) of the fast version. The slow version is merged into the backbone, guiding the model toward flatter, more generalizable minima.
  3. Cascading Noisy Tuning: To escape local optima, task-aware random noise is injected into the backbone before training each new expert. The noise scale is dynamically tied to the standard deviation of the current "slow" LoRA weights.
The LoRASC training pipeline involving fast training, slow merging, and weight updates.

Crucially, because LoRA modules are re-parameterizable, all experts are merged into the original weights post-training, resulting in zero additional inference latency.

Experimental Evaluation

The authors evaluated LoRASC across a wide array of benchmarks, including SuperGLUE (NLU), GSM8K (math reasoning), SQuAD (QA), and ImageNet-1K (CV), alongside various robustness benchmarks (ImageNet-A/C/R).

Key Results

LoRASC consistently outperformed vanilla LoRA, LoRA+, and DoRA. Notable performance gains include:

  • Math Reasoning: On GSM8K, LoRASC improved LoRA's performance from 19.5% to 27.5%.
  • Instruction Following: On the BBH benchmark, scores rose from 13.41 to 17.07.
  • Vision Robustness: On ImageNet-R, top-1 accuracy improved from 87.1% to 88.7%.
Comparative performance of LoRASC components against baseline PEFT methods.

Discussion

A core insight of LoRASC is that rank is cumulative. By merging and re-initializing, the model bypasses the low-rank bottleneck, accumulating a higher-rank transformation while maintaining low GPU memory usage. The slow-fast update acts as a powerful regularizer, preventing the catastrophic forgetting or overfitting often seen when models chase gradients too aggressively. Furthermore, the self-scaling perturbations make the noise "task-aware," adjusting intensity based on the magnitude of learned updates.

Conclusion

LoRASC represents a significant step forward in making PEFT more robust and expressive. While it introduces new hyperparameters like the EMA decay rate ($\alpha$) and noise intensity ($\lambda$), its stability across epochs and ranks makes it a more reliable choice than standard LoRA. Future work will explore extending cascaded learning to multi-task and multi-domain adaptation scenarios.

Summary updated

Abstract

Efficient fine-tuning plays a fundamental role in modern large models, with low-rank adaptation emerging as a particularly promising approach. However, the existing variants of LoRA are hampered by limited expressiveness, a tendency to overfit, and sensitivity to hyperparameter settings. This paper presents LoRA Slow Cascade Learning (LoRASC), an innovative technique designed to enhance LoRA’s expressiveness and generalization capabilities while preserving its training efficiency. Our approach augments expressiveness through a cascaded learning strategy that enables a mixture-of-low-rank adaptation, thereby increasing the model’s ability to capture complex patterns. Additionally, we introduce a slow-fast update mechanism and cascading noisy tuning to bolster generalization. The extensive experiments on various language and vision datasets, as well as robustness benchmarks, demonstrate that the proposed method not only significantly outperforms existing baselines, but also mitigates overfitting, enhances model stability, and improves OOD robustness.
BibTeX
@inproceedings{li2024lorasc,
  author = {Siwei Li and Yifan Yang and Yifei Shen and Fangyun Wei and Zongqing Lu and Lili Qiu and Yuqing Yang},
  title = {{LoRASC: Expressive and Generalizable Low-rank Adaptation for Large Models via Slow Cascaded Learning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.748},
}
Figure