Training Neural Networks from Scratch with Parallel Low-Rank Adapters

Minyoung Huh, Brian Cheung, Jeremy Bernstein, Phillip Isola, Pulkit AgrawalSubmission to ICLR 2024 · Rejected

TL;DR

Low-Rank Adaptation (LoRA) is ubiquitous for parameter-efficient fine-tuning, but standard LoRA fails when training deep neural networks from scratch due to rigid low-rank constraints. This paper introduces LoRA-the-Explorer (LTE), a distributed bi-level optimization framework that trains multiple independent LoRA heads in parallel with infrequent merging. LTE enables pre-training models from scratch while fitting models roughly $3\times$ larger onto memory-constrained GPUs and cutting inter-node communication bandwidth in half.

What Problem Does This Paper Address?

Pre-training modern neural architectures like Vision Transformers (ViTs) and Large Language Models (LLMs) requires massive compute, memory, and high-bandwidth interconnects (e.g., InfiniBand). Standard Distributed Data Parallel (DDP) training consumes substantial GPU memory just storing optimizer states (e.g., AdamW stores two states per parameter) and requires all-reducing full-rank gradients at every single step.

While parameter-efficient fine-tuning methods like LoRA dramatically reduce optimizer memory and communication, standard LoRA fails when training from scratch. The authors observe that the effective rank of gradients grows steadily throughout pre-training; a single static low-rank adapter simply cannot capture the trajectory of full-rank model updates.

Methods

To bridge this gap, the authors propose LoRA-the-Explorer (LTE):

  • Parallel Multi-Head LoRA: For every linear layer $W$, LTE assigns independent rank-$r$ adapter heads ($B_n, A_n$) to $N$ parallel workers, each processing distinct data batches. The combined updates sum to an effective rank of up to $N \times r$.
  • Local Optimization with Infrequent Merges: Each worker updates its local LoRA parameters using a local optimizer for $T$ iterations (e.g., $T = 10$ or $20$) without synchronizing intermediate gradients or optimizer states.
  • Periodic Aggregation & Reset: Every $T$ steps, workers synchronize adapter updates. The averaged low-rank update $\Delta W = \frac{s}{N} \sum_{n=1}^N B_n A_n$ is merged directly into the base weights $W$. Matrix $B_n$ is reset to zero for the next cycle.
  • Key Implementation Recipe:
  • Keep $A_n$ and Optimizer States: Counter-intuitively, resetting $A_n$ or clearing the optimizer state between merge cycles discards learned directional features and slows convergence. Keeping them preserves subspace exploration without divergence.
  • High Scaling ($s$), Low Learning Rate ($\eta$): Using large scale factors ($s \approx 64\text{--}256$) with smaller learning rates introduces a spectral alignment dynamic between $A$ and $B$, which is essential for stable pre-training.
  • Quantized Base Weights: The base model $W$ can reside in low precision (e.g., 4-bit) during training, reducing per-device footprint.

Datasets

  • Vision: ImageNet-100 and full ImageNet-1K (primary benchmarks); CIFAR-10, CIFAR-100, STL-10, Caltech-256, and SUN-397.
  • Language: Shakespeare (using MiniGPT) and TinyStories (using GPT-2).
  • Synthetic: Least-squares regression datasets designed with controlled target ranks. (No new datasets were introduced.)

Key Insights and Ideas

  • Multi-Head Decomposition as Full-Rank Approximation: A matrix of rank $d$ can be expressed as a linear combination of lower-rank matrices. Ensuring the product $N \times r$ exceeds the model's widest hidden dimension allows parallel low-rank adapters to reconstruct full-rank trajectories.
  • Natural Subspace Diversity: When initialized with different weights and fed different mini-batches, individual LoRA heads maintain high subspace orthogonality (measured via Grassmannian and cosine distances) throughout training without explicit diversity regularizers.
  • $s$ is Not Just a Learning Rate Multiplier: In adaptive optimizers like AdamW, the LoRA scale factor $s$ does not linearly scale parameter step sizes. Instead, it alters the forward-pass scale and induces a quadratic alignment term ($\propto s^2 \eta$) in the effective gradient update, driving $B$ and $A$ to align spectrally.

Key Results

  • ImageNet-100 (ViT-S): LTE with 32 heads ($r=64, T=10$) achieves 73.73% top-1 accuracy, matching or exceeding full-model baseline training (71.97%), likely aided by the ensembling and regularizing effects of multi-head parameter averaging.
  • ImageNet-1K (ViT-S): LTE matches standard DDP pre-training accuracy (68.0% top-1).
  • Hardware Efficiency: LTE accommodates base models roughly $3\times$ larger on low-memory GPUs using 4-bit quantized base weights ($qM + 3M_{\text{lte}}$ footprint vs. $3M$ for DDP). Furthermore, communicating low-rank weights every $T$ steps reduces total network communication bandwidth by approximately half compared to standard DDP.
  • Generalization Across Architectures: Successfully trains Vision Transformers (ViT-S, ViT-B, ViT-L), MLP-Mixers, and autoregressive language models (nanoGPT / GPT-2).

Limitations and Future Work

  • Convergence Slowdown: Matching baseline accuracy on ImageNet-1K requires roughly 40% more epochs (420 epochs vs. 300 epochs), largely due to gradient noise from smaller per-head mini-batches and stale parameter estimates during late-stage training.
  • Throughput Overhead: Low-precision quantizations (such as QLoRA) currently incur a ~20% per-iteration computation penalty.
  • Unresolved Dynamics: The authors leave several questions open: scheduling dynamic rank allocations across heads, handling heterogeneous hardware configurations, and incorporating advanced federated/stitching merge algorithms to permit larger local intervals ($T \gg 20$).

Summary updated

Abstract

The scalability of deep learning applications is fundamentally constrained by compute, memory, and communication. While low-rank adaptation (LoRA) has reduced these costs for model fine-tuning, its application to model pre-training remain largely unexplored. This paper examines the extension of LoRA to model pre-training, identifying the constraints and limitations inherent to standard LoRA in the context of pre-training. We introduce LoRA-the-Explorer (LTE), a novel bi-level optimization algorithm, to facilitate parallel training of multiple low-rank heads across compute nodes, minimizing the necessity for frequent synchronization. Our methodology involves rigorous experimentation on vision transformers using ImageNet100, demonstrating that LTE is competitive with standard distributed training methodologies. Initial scalability tests on ImageNet1k show that LTE can match standard training performance by leveraging more training iterations.
BibTeX
@misc{huh2024training,
  author = {Minyoung Huh and Brian Cheung and Jeremy Bernstein and Phillip Isola and Pulkit Agrawal},
  title = {{Training Neural Networks from Scratch with Parallel Low-Rank Adapters}},
  year = {2024},
  eprint = {2402.16828},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.16828},
}

Loading related content...

Figure