Abstract
Running Large Language Models (LLMs) locally on edge devices is essential for preserving user privacy, yet it is severely constrained by the limited memory and computational power of individual devices. This paper introduces Jupiter, a collaborative edge AI system designed to pool the resources of multiple idle edge devices—such as smart home hubs, laptops, and smartphones—using a communication-efficient pipeline parallelism approach. By optimizing both the prompt processing (prefill) and token generation (decoding) phases, Jupiter achieves up to a 26.1x reduction in end-to-end latency compared to traditional collaborative frameworks.
Introduction
The demand for local LLM execution is driven by the need to protect sensitive user data from cloud-based processing. However, even modern edge devices often lack the RAM required to host 7B or 13B parameter models. Collaborative edge computing offers a solution by networking trusted local devices via Wi-Fi to share the inference load.
As shown in Figure 1, Jupiter creates a "Collaborative Edge Resource Pool," allowing devices like TVs, smart speakers, and phones to work together. While previous attempts at collaborative inference relied on data center techniques like Tensor Parallelism (TP), these methods often fail in edge environments due to high communication overhead.
The Communication Bottleneck in Edge Parallelism
Existing collaborative systems typically employ Tensor Parallelism (TP) or Sequence Parallelism (SP). In these paradigms, massive amounts of data must be synchronized between devices at nearly every layer of the neural network. On a standard 100Mbps edge network, the communication-to-computation ratio can reach as high as 8.2x, meaning devices spend over 80% of their time waiting for data transfer.
Jupiter addresses this by adopting Pipeline Parallelism (PP). As illustrated in Figure 5, PP slices the LLM horizontally across devices. Each device processes a subset of layers and passes only small intermediate activations to its neighbor, significantly reducing network traffic compared to the frequent synchronizations required by TP and SP.
Proposed Methodology
To maximize the efficiency of Pipeline Parallelism for single-user requests, Jupiter introduces two primary innovations:
1. Intra-Sequence Pipeline Parallelism (Prefill Phase)
Standard pipeline parallelism often suffers from device underutilization when processing a single prompt. Jupiter overcomes this by splitting the input prompt into smaller sub-sequences. By leveraging the causal nature of LLMs—where tokens only attend to previous ones—Jupiter can inject these sub-sequences into the pipeline concurrently. A dynamic programming algorithm is used to profile heterogeneous devices and partition model layers and sequence chunks optimally.
2. Outline-Based Pipeline Parallel Decoding (Decoding Phase)
The autoregressive decoding phase is inherently sequential and difficult to parallelize. Jupiter introduces an "outline-based" approach: the system first prompts the LLM to generate a brief outline of the response. Each point in the outline is then expanded concurrently across the pipeline. For tasks where logical chaining is paramount, Jupiter falls back to Speculative Decoding, using the model's own heads to draft and verify multiple tokens simultaneously.
Evaluation and Results
Jupiter was evaluated using datasets such as LiMA, Vicuna-80, and WizardLM on clusters of NVIDIA Jetson devices. Key findings include:
- Latency Reduction: Jupiter achieved up to a 26.1x reduction in end-to-end latency compared to TP-based methods and up to 3.3x compared to SP-based methods.
- Decoding Speedup: The combination of speculative and outline-based decoding provided a 3.9x speedup over naive sequential generation.
- Quality Preservation: Evaluation via the FastChat framework (using GPT-4o as a judge) confirmed that outline-based generation maintains high response quality across most categories.
Conclusion and Future Work
Jupiter demonstrates that pipeline parallelism, combined with clever sequence-level optimizations, makes collaborative LLM inference viable on resource-constrained edge networks. While the outline-based approach faces challenges with strict logical tasks like mathematics, the system's modular design allows for flexible fallback strategies. Future research will focus on enhancing parallelization for complex reasoning tasks without sacrificing logical consistency.
Summary updated