Pluto and Charon: A Time and Memory Efficient Collaborative Edge AI Framework for Personal LLMs Fine-tuning

Bei Ouyang, Shengyuan Ye, Liekang Zeng, Tianyi Qian, Jingyi Li, Xu Chen

Introduction

The rapid advancement of Large Language Models (LLMs) has paved the way for Intelligent Personal Assistants (IPAs) capable of sophisticated reasoning and task execution. However, to ensure user privacy and provide deep personalization, these models must be fine-tuned on local, user-specific data. This requirement introduces a significant technical challenge: the "resource wall." While Parameter-Efficient Fine-Tuning (PEFT) techniques like LoRA reduce the number of trainable parameters, they still necessitate a full forward and backward pass through the massive, frozen LLM backbone. For resource-constrained edge devices—such as smart home hubs, tablets, or smartphones—the resulting memory footprint and computational latency are often prohibitive.

This paper presents PAC+ (Pluto and Charon), a novel collaborative edge AI framework designed to bridge the gap between the heavy requirements of LLM fine-tuning and the limited capabilities of edge hardware. By employing an algorithm-system co-design, PAC+ achieves significant speedups and memory reductions, making in-situ personalization a reality for the edge.

Methodology

PAC+ addresses the "memory wall" and "latency wall" through four primary innovations that decouple the training process from the resource-intensive LLM backbone.

1. Parallel Adapters

Traditional adapters are typically inserted sequentially within transformer layers, forcing the system to backpropagate gradients through the entire model. PAC+ introduces Parallel Adapters, a lightweight proxy network that runs alongside the frozen backbone. During fine-tuning, only this side-network is updated. This creates a "gradient highway" that completely bypasses the main LLM during the backward pass, drastically reducing the computational overhead.

2. Activation Caching

Because the LLM backbone remains frozen during the fine-tuning process, its intermediate outputs (activations) for a specific input sequence are invariant across training epochs. PAC+ exploits this invariance by caching these activations during the first epoch. In all subsequent epochs, the backbone is powered down, and the system trains the Parallel Adapters using only the cached data. This "train once, reuse many" approach eliminates the need for repeated forward passes through the backbone, which is the most expensive part of the model.

3. Collaborative Edge Resource Pool

Recognizing that a single edge device (e.g., a Jetson Nano) may still lack the RAM to host a large model even with optimizations, PAC+ aggregates multiple idle devices within a local network into a single virtual resource pool. This allows the framework to distribute the model's memory and compute requirements across a cluster of devices, such as a smart fridge, a tablet, and a home hub working in tandem.

4. Heterogeneity-aware Planning

Edge environments are inherently heterogeneous, consisting of devices with varying compute power and memory capacities. PAC+ utilizes a dynamic programming-based planner to optimize throughput. The planner determines the ideal hybrid of Data Parallelism and Pipeline Parallelism, partitioning the model across the available devices based on their specific hardware profiles to prevent bottlenecks.

Experimental Evaluation

The authors evaluated PAC+ using three prominent LLMs: T5-Base (0.25B parameters), T5-Large (0.74B parameters), and BART-Large (0.41B parameters). Performance was benchmarked across four tasks from the GLUE benchmark: SST-2 (Sentiment Analysis), MRPC and STS-B (Similarity/Paraphrase), and QNLI (Natural Language Inference).

Performance Metrics

  • Speedup: PAC+ achieved up to a 9.7× end-to-end speedup compared to existing collaborative edge training systems like Asteroid and HetPipe.
  • Memory Reduction: Compared to mainstream PEFT algorithms, PAC+ reduced the peak memory footprint by up to 88.16%. This reduction is critical for enabling models like T5-Large to run on devices that would otherwise trigger Out-of-Memory (OOM) errors.
  • Accuracy: Despite aggressive optimizations and the use of INT8 and INT4 quantization, PAC+ maintained model performance comparable to full-parameter fine-tuning, with negligible differences in GLUE scores.
  • Scalability: The system demonstrated strong scaling properties; as more devices were added to the local resource pool, throughput increased significantly while the memory pressure on individual devices decreased.

Discussion and Limitations

While PAC+ represents a major step forward for edge AI, certain trade-offs exist. The Activation Caching mechanism requires local storage space. Although this typically accounts for less than 1% of modern mobile storage, it could become a factor for exceptionally long sequences or massive datasets. Additionally, the Initial Epoch Latency remains high because the full backbone must run once to populate the cache. Finally, the system's efficiency is dependent on Network Stability; high jitter or packet loss in a local area network (LAN) could disrupt the pipeline parallelism.

Conclusion

PAC+ demonstrates that parameter efficiency does not automatically equate to resource efficiency. By rethinking the interaction between adapters and the LLM backbone, and by leveraging the collective power of local edge devices, PAC+ breaks the resource wall. This framework enables the deployment of truly personal, private, and high-performance AI assistants that can learn and evolve directly on the user's hardware.

Summary updated

BibTeX
@inproceedings{ouyang2024pluto,
  author = {Bei Ouyang and Shengyuan Ye and Liekang Zeng and Tianyi Qian and Jingyi Li and Xu Chen},
  title = {{Pluto and Charon: A Time and Memory Efficient Collaborative Edge AI Framework for Personal LLMs Fine-tuning}},
  booktitle = {ICPP},
  year = {2024},
  doi = {10.1145/3673038.3673043},
}
Figure