TL;DR
This paper introduces FwdLLM, a novel federated learning framework that enables efficient fine-tuning of large language models (LLMs) on mobile devices. By replacing traditional backpropagation with a "perturbed inference" approach, FwdLLM significantly reduces memory and computational costs, making it possible to train billion-parameter LLMs on consumer-grade hardware for the first time.
What Problem Does This Paper Address?
Large Language Models (LLMs) are revolutionizing various fields, but their immense size and computational demands pose a significant challenge for deployment and fine-tuning on resource-constrained mobile devices. Federated Learning (FL) offers a privacy-preserving way to fine-tune these models using data distributed across many devices. However, existing FL approaches for LLMs, often referred to as FedLLM, are hampered by three main issues:
- Memory Wall: LLMs require substantial memory to store intermediate computations (activations and gradients) during training, often exceeding the capacity of mobile devices.
- Incompatibility with Mobile Accelerators: Mobile System-on-Chips (SoCs) have powerful Neural Processing Units (NPUs) optimized for inference, but they typically lack the necessary support for backpropagation-based training operations.
- Limited Device Scalability: Traditional FL methods struggle to effectively leverage a large number of participating devices, leading to slow convergence.
These challenges make it impractical to fine-tune large LLMs on typical mobile phones.
Methods
FwdLLM tackles these problems by introducing a BP-free (backpropagation-free) federated learning framework. The core idea is to replace the computationally intensive backpropagation with a more efficient "perturbed inference" method. Instead of calculating exact gradients, each mobile device performs inferences on the model with slightly perturbed weights. By comparing the output deviations from these perturbed inferences to the ground truth, FwdLLM can estimate a gradient that directs the model towards global optima.
Key innovations in FwdLLM include:
- Parameter-Efficient Fine-Tuning (PEFT) Integration: FwdLLM combines BP-free training with PEFT methods like LoRa and Adapter. This is crucial because the computational complexity of BP-free methods scales with the number of trainable parameters, not the total model size, aligning perfectly with PEFT's goal of reducing trainable parameters.
- Variance-Controlled Perturbation Pacing: FwdLLM introduces an adaptive strategy to manage the number of perturbations. It monitors the variance of incoming gradients from clients and dynamically adjusts the pacing to ensure efficient convergence. This avoids the need for manual tuning of this critical hyperparameter.
- Discriminative Perturbation Sampling: Instead of randomly sampling perturbations, FwdLLM selectively generates perturbations that are more likely to contribute to model convergence. This is achieved by estimating the direction of the true gradient and prioritizing perturbations that are more aligned with it, significantly improving efficiency.
Datasets
The paper evaluates FwdLLM on four popular NLP datasets:
- AGNEWS: A news classification dataset.
- YAHOO: A topic classification dataset.
- YELP-Polarity (YELP-P): A dataset for predicting sentiment from restaurant reviews.
- SQuAD-v1.1: A question answering dataset.
Key Insights and Ideas
- Backpropagation is the Bottleneck: The paper strongly argues that backpropagation is the primary culprit behind the inefficiency of FedLLM on mobile devices, hindering memory efficiency, NPU utilization, and scalability.
- Perturbed Inferences are Powerful: The core insight is that by carefully perturbing model weights and observing inference outputs, one can derive useful gradient information without the overhead of backpropagation.
- PEFT and BP-Free are Synergistic: Combining PEFT with BP-free methods is a natural fit, as both aim to reduce computational burden, and their combined effect is greater than the sum of their parts for LLM fine-tuning.
- Adaptive Strategies are Crucial: Dynamic pacing of perturbations and discriminative sampling are not just optimizations but essential components for making BP-free FL practical and efficient for large models.
- LLMs on COTS Devices are Now Possible: FwdLLM, especially when combined with quantization, demonstrates that training billion-parameter LLMs like LLaMA on standard mobile phones is no longer a distant dream.
Key Results
- Massive Speedup: FwdLLM reduces training time by up to 217.3x compared to full-model fine-tuning. Compared to other competitive baselines, it achieves an average speedup of 10.6x.
- Significant Memory Reduction: FwdLLM reduces memory footprint by up to 14.6x compared to baselines.
- NPU Utilization: FwdLLM can effectively leverage mobile NPUs, which are typically only used for inference, leading to substantial performance gains.
- Billion-Parameter LLM Training: For the first time, FwdLLM enables federated fine-tuning of a billion-parameter model (LLaMA-7B) on COTS mobile devices, achieving target accuracy in just minutes with INT4 quantization.
- Scalability: FwdLLM shows excellent scalability with the number of clients, with convergence time significantly decreasing as more devices participate.
Limitations and Future Work
- BP-Free Generalization: While FwdLLM shows promise, the paper acknowledges that BP-free methods have historically been validated on smaller models. The effectiveness on extremely large LLMs still requires further exploration.
- Hyperparameter Tuning: Although FwdLLM introduces adaptive strategies, some hyperparameters (like the variance threshold for pacing) still require careful selection, though the paper suggests empirically effective ranges.
- NPU Support for LLaMA: The paper notes that LLaMA is not yet supported by mobile NPUs, and its speedup on NPU is emulated based on other models. Future work could involve enabling direct NPU support for such models.
- Further Optimization: The authors suggest that further research into optimizing the perturbation generation and sampling process, as well as exploring more advanced PEFT techniques, could lead to even greater efficiency.
Summary updated