Training Neural Networks from Scratch with Parallel Low-Rank Adapters

Minyoung Huh, Brian Cheung, Jeremy Bernstein, Phillip Isola, Pulkit Agrawal

Abstract

While Low-Rank Adaptation (LoRA) has revolutionized model fine-tuning, it has historically failed to match the performance of standard training when starting from scratch. This paper introduces LoRA-the-Explorer (LTE), a novel algorithm that enables full-rank pre-training by using multiple low-rank adapters in parallel. This approach allows researchers to train large models on memory-constrained hardware with significantly reduced communication overhead, effectively bridging the gap between parameter-efficient methods and full-scale pre-training.

What Problem Does This Paper Address?

Training state-of-the-art deep learning models from scratch is a resource-heavy endeavor, often requiring massive GPU clusters and high-bandwidth interconnects. While Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA have made adapting models accessible by only updating a small fraction of parameters, pre-training with these methods has remained an elusive goal.

The core challenge is the "rank constraint." Standard LoRA updates are low-rank by design. However, during pre-training, the gradients required to learn a model from scratch are inherently high-rank. A single LoRA head simply doesn't have the expressive capacity to capture the complex weight changes needed in the early stages of training.

Test loss comparison showing the performance gap between full-rank training and standard low-rank LoRA during pre-training.

As shown in the figure above, models trained from scratch with vanilla low-rank LoRA consistently underperform compared to standard full-parameter training.

Methods

The authors propose LoRA-the-Explorer (LTE), a bi-level optimization algorithm designed to bridge this performance gap. The approach is built on three key steps:

  1. Parallelization: Instead of one LoRA head, LTE initializes $N$ different LoRA heads across multiple computing nodes.
  2. Independent Exploration: Each worker trains its unique LoRA head independently for $T$ iterations using its own partition of data. This allows different heads to explore distinct subspaces of the parameter space.
  3. Delayed Merging: Periodically, these low-rank updates are synchronized, averaged, and merged back into the main weights ($W$).
The LTE optimization cycle: independent local training of LoRA heads followed by a global commit and merge step.

Crucially, the authors found that the sum of multiple low-rank updates can effectively approximate a high-rank update. To further optimize for memory, the main weights $W$ can be stored in low-precision (e.g., 4-bit quantization), while only the small LoRA adapters are kept in high precision.

Key Insights and Ideas

  • The Power of Parallelism: A single rank-8 update is insufficient, but 32 parallel rank-8 updates (effectively rank-256 when merged) can recover the dynamics of full-rank training.
  • Diversity Prevents Collapse: By using different data mini-batches and different initializations for each LoRA head, the authors ensure the heads don't "collapse" into the same representation. They remain orthogonal, maximizing the information captured during each training cycle.
  • Scaling vs. Learning Rate: The authors show that the LoRA scaling factor ($s$) affects forward pass dynamics quadratically in attention maps. Using a large $s$ with a lower learning rate ($η$) is critical for stable pre-training.
  • Initialization Matters: Standard Kaiming initialization performs poorly for the rectangular matrices used in LoRA. Using semi-orthogonal initialization significantly improved convergence.

Key Results

  • Performance Parity: LTE successfully matched the top-1 accuracy of standard full-model pre-training on ImageNet-1k (68% for ViT-S).
  • Memory Efficiency: Because the main weights can be quantized and only LoRA parameters are optimized, LTE can fit models 3x larger on the same GPU hardware compared to standard Distributed Data Parallel (DDP) training.
  • Communication Savings: LTE reduces communication bandwidth requirements by roughly 50% because workers only need to sync small LoRA parameters rather than full-model gradients.
  • The Convergence Tax: The main trade-off is speed. LTE requires approximately 40% more training samples to reach the same accuracy as standard training, likely due to the noise introduced by stale parameter estimates between merges.

Limitations and Future Work

While LTE provides a promising proof-of-concept, it struggles to match the final 10% of convergence speed seen in standard optimization. Future work could explore dynamically changing the number of heads or the rank ($r$) as training progresses, or employing more sophisticated merging techniques like TIES-merging to further reduce the "convergence tax."

In summary, LTE demonstrates that the "wisdom of the crowd"—in the form of parallel low-rank adapters—can finally bring the memory benefits of LoRA to the world of model pre-training.

Summary updated

Abstract

The scalability of deep learning models is fundamentally limited by computing resources, memory, and communication. Although methods like low-rank adaptation (LoRA) have reduced the cost of model finetuning, its application in model pre-training remains largely unexplored. This paper explores extending LoRA to model pre-training, identifying the inherent constraints and limitations of standard LoRA in this context. We introduce LoRA-the-Explorer (LTE), a novel bi-level optimization algorithm designed to enable parallel training of multiple low-rank heads across computing nodes, thereby reducing the need for frequent synchronization. Our approach includes extensive experimentation on vision transformers using various vision datasets, demonstrating that LTE is competitive with standard pre-training.
BibTeX
@misc{huh2024training,
  author = {Minyoung Huh and Brian Cheung and Jeremy Bernstein and Phillip Isola and Pulkit Agrawal},
  title = {{Training Neural Networks from Scratch with Parallel Low-Rank Adapters}},
  year = {2024},
  eprint = {2402.16828},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.16828},
}
Figure