Resource-Efficient Personal Large Language Models Fine-Tuning with Collaborative Edge Computing

Bei Ouyang, Shengyuan Ye, Liekang Zeng, Tianyi Qian, Jingyi Li, Xu Chen, Chu, Xiaowen, Xing, Guoliang, Chen, Xu

Abstract

Large language models (LLMs) have unlocked a plethora of powerful applications at the network edge, such as intelligent personal assistants. Data privacy and security concerns have prompted a shift towards edge-based fine-tuning of personal LLMs, away from cloud reliance. However, this raises issues of computational intensity and resource scarcity, hindering training efficiency and feasibility. While current studies investigate parameter-efficient fine-tuning (PEFT) techniques to mitigate resource constraints, our analysis indicates that these techniques are not sufficiently resource-efficient for edge devices. To tackle these challenges, we propose Pluto and Charon (PAC), a time and memory efficient collaborative edge AI framework for personal LLMs fine-tuning. PAC breaks the resource wall of personal LLMs fine-tuning with a sophisticated algorithm-system co-design. (1) Algorithmically, PAC implements a personal LLMs fine-tuning technique that is efficient in terms of parameters, time, and memory. It utilizes Parallel Adapters to circumvent the need for a full backward pass through the LLM backbone. Additionally, an activation cache mechanism further streamlining the process by negating the necessity for repeated forward passes across multiple epochs. (2) Systematically, PAC leverages edge devices in close proximity, pooling them as a collective resource for in-situ personal LLMs fine-tuning, utilizing a hybrid data and pipeline parallelism to orchestrate distributed training. The use of the activation cache eliminates the need for forward pass through the LLM backbone, enabling exclusive fine-tuning of the Parallel Adapters using data parallelism. Extensive evaluation based on prototype implementation demonstrates that PAC remarkably outperforms state-of-the-art approaches, achieving up to 8.64x end-to-end speedup and up to 88.16% reduction in memory footprint.

Introduction

The proliferation of Large Language Models (LLMs) has catalyzed the development of sophisticated edge applications, most notably intelligent personal assistants. To ensure data privacy and security, there is a growing imperative to move LLM fine-tuning from centralized cloud servers to the network edge. This transition enables "Personal LLMs" to learn from sensitive user data locally. However, edge devices—such as smart home hubs or mobile phones—face a significant "resource wall." Fine-tuning even moderately sized models is computationally intensive and memory-demanding, often exceeding the physical limits of individual edge nodes.

Overview of the Personal LLM ecosystem, from fine-tuning with private data to smart home serving.

The Resource Wall of Edge Fine-Tuning

While Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA or standard Adapters reduce the number of trainable parameters, they do not necessarily solve the resource problem for edge devices. A critical bottleneck remains: standard PEFT still requires a full "backward pass" through the entire frozen backbone to calculate gradients for the trainable layers. As shown in the computational analysis, the backward pass remains a dominant factor in total TFLOPs, often exceeding the capacity of low-power hardware like the NVIDIA Jetson Nano.

Comparison of TFLOPs for forward and backward passes across different models and PEFT methods.

The PAC+ Framework

To overcome these limitations, we introduce PAC+, a collaborative edge computing framework designed for resource-heavy LLM fine-tuning. PAC+ employs a dual-pronged approach involving algorithm-system co-design:

1. Parallel Adapters and Activation Caching

Unlike traditional adapters inserted sequentially within the backbone, PAC+ utilizes Parallel Adapters. These function as a "side-car" architecture where gradients flow only through the lightweight adapters, completely bypassing the massive backbone during backpropagation.

Furthermore, PAC+ introduces Activation Caching. Since the backbone remains frozen, its output for any given training input is deterministic. By running the backbone once during the initial epoch and caching the resulting activations, PAC+ can skip the backbone entirely in subsequent epochs. This effectively "deletes" the backbone from the training loop after the first pass, drastically reducing both compute and memory requirements.

2. Collaborative Edge Orchestration

PAC+ treats a local network of devices as a unified resource pool. It utilizes a Hybrid Parallelism strategy, combining Data Parallelism (DP) and Pipeline Parallelism (PP). A heterogeneity-aware planning algorithm, powered by dynamic programming, optimizes the distribution of model layers across devices based on their specific memory and compute profiles. To further minimize footprint, the backbone is stored using 4-bit or 8-bit quantization, while adapters maintain 32-bit precision for accuracy.

Key Results

Experimental evaluations using T5 and BART models on GLUE benchmark tasks demonstrate the efficacy of PAC+:

  • Performance Gains: PAC+ achieves up to a 9.7× end-to-end speedup compared to existing collaborative edge training systems like Eco-FL.
  • Memory Efficiency: The framework reduces peak memory footprint by up to 88.16%. This allows a T5-Large model (0.74B parameters), which typically requires ~11GB of RAM, to be fine-tuned on a device with only 4GB of RAM.
  • Accuracy Preservation: Despite aggressive optimizations, PAC+ maintains accuracy within 0.1–0.3% of full-parameter fine-tuning and standard LoRA.

Conclusion and Future Work

PAC+ successfully breaks the resource wall for edge-based LLM fine-tuning by decoupling the trainable components from the backbone and leveraging collaborative device pools. Future research will focus on scaling these techniques to larger models (e.g., 7B+ parameters) and addressing the challenges of dynamic device churn in real-world smart home environments.

Summary updated

Abstract

Large language models (LLMs) have unlocked a plethora of powerful applications at the network edge, such as intelligent personal assistants. Data privacy and security concerns have prompted a shift towards edge-based fine-tuning of personal LLMs, away from cloud reliance. However, this raises issues of computational intensity and resource scarcity, hindering training efficiency and feasibility. While current studies investigate parameter-efficient fine-tuning (PEFT) techniques to mitigate resource constraints, our analysis indicates that these techniques are not sufficiently resource-efficient for edge devices. To tackle these challenges, we propose Pluto and Charon (PAC), a time and memory efficient collaborative edge AI framework for personal LLMs fine-tuning. PAC breaks the resource wall of personal LLMs fine-tuning with a sophisticated algorithm-system co-design. (1) Algorithmically, PAC implements a personal LLMs fine-tuning technique that is efficient in terms of parameters, time, and memory. It utilizes Parallel Adapters to circumvent the need for a full backward pass through the LLM backbone. Additionally, an activation cache mechanism further streamlining the process by negating the necessity for repeated forward passes across multiple epochs. (2) Systematically, PAC leverages edge devices in close proximity, pooling them as a collective resource for in-situ personal LLMs fine-tuning, utilizing a hybrid data and pipeline parallelism to orchestrate distributed training. The use of the activation cache eliminates the need for forward pass through the LLM backbone,enabling exclusive fine-tuning of the Parallel Adapters using data parallelism. Extensive evaluation based on prototype implementation demonstrates that PAC remarkably outperforms state-of-the-art approaches, achieving up to 8.64x end-to-end speedup and up to 88.16% reduction in memory footprint.
BibTeX
@inproceedings{ouyang2024resource,
  author = {Bei Ouyang and Shengyuan Ye and Liekang Zeng and Tianyi Qian and Jingyi Li and Xu Chen and Chu, Xiaowen and Xing, Guoliang and Chen, Xu},
  title = {{Resource-Efficient Personal Large Language Models Fine-Tuning with Collaborative Edge Computing}},
  booktitle = {ICPP},
  year = {2024},
  doi = {10.1145/3673038.3673043},
}
Figure