Abstract
LoRASC (Low-Rank Adaptation via Slow Cascaded Learning) is a novel fine-tuning framework designed to enhance the expressiveness and generalization of Low-Rank Adaptation (LoRA) without introducing inference overhead. By iteratively training and merging cascaded low-rank experts through a slow-fast update mechanism, LoRASC achieves state-of-the-art performance across diverse language and vision benchmarks while effectively mitigating common overfitting issues associated with standard Parameter-Efficient Fine-Tuning (PEFT) methods.
Introduction
Fine-tuning massive foundation models is computationally demanding, leading to the widespread adoption of PEFT methods like LoRA. LoRA reduces trainable parameters by adding small, low-rank matrices to model layers. However, LoRA faces three primary bottlenecks: limited expressiveness due to low-rank constraints, a tendency to overfit in low-data regimes, and high sensitivity to hyperparameters like rank and learning rate. LoRASC addresses these limitations by rethinking the scheduling and update mechanisms of low-rank modules during training.
Methodology
LoRASC introduces three synergistic components to improve adaptation:
- Cascading LoRA Learning: Instead of a single static LoRA module, LoRASC trains a new "expert" module every epoch. Each expert is merged into the backbone weights at the epoch's end, and a new one is initialized to learn from the remaining residuals. This allows the model to accumulate a higher total rank over time without increasing the memory footprint of any single step.
- Slow-Fast Update Mechanism: Inspired by Stochastic Weight Averaging (SWA), LoRASC maintains a "fast" expert that learns directly from data and a "slow" expert, which is an Exponential Moving Average (EMA) of the fast version. The slow version is merged into the backbone, guiding the model toward flatter, more generalizable minima.
- Cascading Noisy Tuning: To escape local optima, task-aware random noise is injected into the backbone before training each new expert. The noise scale is dynamically tied to the standard deviation of the current "slow" LoRA weights.
Crucially, because LoRA modules are re-parameterizable, all experts are merged into the original weights post-training, resulting in zero additional inference latency.
Experimental Evaluation
The authors evaluated LoRASC across a wide array of benchmarks, including SuperGLUE (NLU), GSM8K (math reasoning), SQuAD (QA), and ImageNet-1K (CV), alongside various robustness benchmarks (ImageNet-A/C/R).
Key Results
LoRASC consistently outperformed vanilla LoRA, LoRA+, and DoRA. Notable performance gains include:
- Math Reasoning: On GSM8K, LoRASC improved LoRA's performance from 19.5% to 27.5%.
- Instruction Following: On the BBH benchmark, scores rose from 13.41 to 17.07.
- Vision Robustness: On ImageNet-R, top-1 accuracy improved from 87.1% to 88.7%.
Discussion
A core insight of LoRASC is that rank is cumulative. By merging and re-initializing, the model bypasses the low-rank bottleneck, accumulating a higher-rank transformation while maintaining low GPU memory usage. The slow-fast update acts as a powerful regularizer, preventing the catastrophic forgetting or overfitting often seen when models chase gradients too aggressively. Furthermore, the self-scaling perturbations make the noise "task-aware," adjusting intensity based on the magnitude of learned updates.
Conclusion
LoRASC represents a significant step forward in making PEFT more robust and expressive. While it introduces new hyperparameters like the EMA decay rate ($\alpha$) and noise intensity ($\lambda$), its stability across epochs and ranks makes it a more reliable choice than standard LoRA. Future work will explore extending cascaded learning to multi-task and multi-domain adaptation scenarios.
Summary updated