LinguaLinked: Distributed Large Language Model Inference on Mobile Devices

Junchen Zhao, Yurun Song, Simeng Liu, Ian G. Harris, Sangeetha Abdu JyothiACL 2024

TL;DR

This paper introduces LinguaLinked, a novel system for running Large Language Models (LLMs) on mobile devices by distributing inference tasks across multiple "trusted" devices. It tackles the significant memory and computational challenges of LLMs on resource-constrained mobile hardware, enabling efficient and private on-device AI.

What Problem Does This Paper Address?

Large Language Models (LLMs) like GPT-3 and LLaMA have revolutionized natural language processing, achieving state-of-the-art results in tasks like text generation and translation. However, their massive size (billions of parameters) makes deploying them directly on mobile devices a significant hurdle due to limited memory and processing power. Traditional server-based inference, while feasible, raises privacy concerns and can introduce latency due to data transmission. While distributed inference has been explored for smaller models (e.g., in computer vision), applying it to LLMs on mobile devices presents unique challenges: managing diverse device capabilities, handling complex data dependencies between model segments, and adapting to dynamic resource availability.

Methods

LinguaLinked employs a three-pronged approach to enable distributed LLM inference on mobile devices:

  • Optimized Model Assignment: LLMs are first converted into computational graphs and then segmented into smaller, independent subgraphs. A primary optimizer, formulated as a linear programming problem, assigns these subgraphs to devices based on their computational characteristics and device capabilities, aiming to minimize computation and data transmission while respecting memory constraints.
  • Runtime Load Balancing: A secondary optimizer refines the initial assignment by dynamically reallocating "movable" sub-modules across devices to balance the load and prevent bottlenecks. This is crucial for handling varying device performance and ensuring responsiveness.
  • Optimized Communication: LinguaLinked uses a decentralized, ring-structured communication approach where devices sequentially process and forward data. It leverages ZeroMQ with a ROUTER-DEALER pattern for efficient, asynchronous communication. To further reduce latency and data transfer, a "residual communication" strategy is introduced, allowing direct transmission of necessary data to target devices rather than passing along intermediate data that isn't immediately needed. Multi-threading is also employed to enhance CPU efficiency and parallel processing.

Datasets

The paper evaluates its system using text generation tasks on the Wikitext-2 dataset with 100 randomly selected samples.

Key Insights and Ideas

  • Decentralization is Key: By distributing LLM inference across multiple mobile devices, LinguaLinked bypasses the limitations of single-device processing and addresses privacy concerns inherent in cloud-based solutions.
  • Heterogeneity is a Feature, Not a Bug: The system is designed to handle devices with diverse capabilities, using optimized assignment and load balancing to leverage the strengths of each device.
  • Residual Communication for Efficiency: Moving beyond purely sequential data flow, the residual communication strategy significantly reduces unnecessary data transfer, leading to faster inference.
  • Larger Models Benefit More: The performance gains from LinguaLinked's optimizations are more pronounced with larger LLMs, indicating its scalability and effectiveness for the most demanding AI tasks.

Key Results

LinguaLinked demonstrates significant performance improvements:

  • Single-threaded inference: Achieved an acceleration of approximately 1.11x to 1.61x compared to a baseline, for both quantized and full-precision models.
  • Multi-threaded inference: Further boosted performance, achieving acceleration rates of 1.73x to 2.65x.
  • Runtime load balancing: Provided an overall inference acceleration of 1.29x to 1.32x.
  • Model Assignment Optimization: For int8 quantized models, throughput improvements were notable, with BLOOM 3b achieving a 1.61x improvement. Full-precision models also saw improvements, with BLOOM 3b showing a 1.32x speed-up.
  • Load Balancer Micro-benchmarking: Showed a significant reduction in inference latency, with improvements of around 30% in acceleration for the BLOOM 1.7b model.

Limitations and Future Work

The authors acknowledge several limitations:

  • Overheads: Load balancing introduces some overhead, and current hardware and software frameworks have constraints.
  • GPU Acceleration: The current evaluation focuses on CPU-only support, anticipating future enhancements with GPU acceleration.
  • Energy Efficiency: Continuous intensive inference, especially with full-precision models, drains battery life and can cause overheating. Future work will focus on incorporating energy-efficient computing strategies and adaptive algorithms.
  • Privacy Model: The current trust model assumes "trusted" mobile devices and doesn't fully address potential side-channel attacks or compromised devices.
  • Standardized Benchmarks: The lack of standardized benchmarks for distributed LLM inference on mobile devices makes direct comparison with prior work challenging.

Future work includes adapting LinguaLinked for distributed fine-tuning on mobile devices, handling multi-modality models, and developing more advanced partitioning and load balancing algorithms that consider battery life and user engagement.

Summary updated

BibTeX
@inproceedings{zhao2024lingualinked,
  author = {Junchen Zhao and Yurun Song and Simeng Liu and Ian G. Harris and Sangeetha Abdu Jyothi},
  title = {{LinguaLinked: Distributed Large Language Model Inference on Mobile Devices}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure