Abstract
Language models (LMs) are machine learning models designed to predict linguistic patterns by estimating the probability of word sequences based on large-scale datasets. While larger datasets typically enhance performance, scalability remains a challenge due to constraints in computational power and resources. Distributed computing strategies offer essential solutions for improving scalability and managing growing demand. Further, the use of sensitive datasets raises significant privacy concerns. This paper presents a survey on distributed solutions for various LMs, including large language models (LLMs), vision language models (VLMs), multimodal LLMs (MLLMs), and small language models (SLMs). We review key advancements across the MLLM pipeline, including distributed training, inference, fine-tuning, and deployment, while identifying contributions, limitations, and future areas of improvement.
Introduction: The Centralization Bottleneck
The AI revolution is currently constrained by what can be termed a "centralization tax." State-of-the-art models like GPT-4 or Llama-3 require thousands of high-end GPUs and massive, often sensitive, datasets. This creates two primary problems: scalability and cost, where training models with hundreds of billions of parameters is prohibitively expensive; and privacy, where training on user data in a central cloud poses significant security risks. This survey explores how to distribute the training, fine-tuning, and inference of these models across a network of diverse, potentially low-power devices while keeping data private and local.
Taxonomy of Distributed Strategies
As a comprehensive survey, this work categorizes the current state-of-the-art across six primary focus areas of decentralization:
- Distributed Training: Utilizing model, data, and pipeline parallelism to split massive workloads.
- Distributed Inference & Optimization: Implementing systems like FastServe that use preemptive scheduling to reduce latency.
- Federated Learning (FL) & Fine-tuning: Employing methods like FedIT and FedKSeed to allow models to learn from decentralized data without it ever leaving the user's device.
- Computing Infrastructures: Leveraging a spectrum of hardware from massive GPU clusters to low-cost edge devices like the NVIDIA Jetson Orin.
- Communication Efficiency: Reducing the "bandwidth tax" of model updates using quantization and gradient compression.
- Mobile Edge Intelligence: Bringing LLM capabilities directly to smartphones and IoT sensors.
The MLLM Pipeline
A critical contribution of this survey is the detailed analysis of the Multimodal Large Language Model (MLLM) pipeline. Unlike text-only models, MLLMs must handle multiple modalities (text, image, audio) and integrate them into a shared embedding space. This requires unique alignment modules and specialized encoders (e.g., CLIP for images) that have distinct computational profiles, complicating the distribution process.
Key Insights and Results
Research indicates a significant shift in the field:
- Communication is the New Compute: In distributed settings, the time spent moving data between devices often exceeds calculation time. Recent research focuses on "smarter communication," such as hiding latency behind computation.
- The Rise of SLMs: Small Language Models (1B–7B parameters) are emerging as the "unsung heroes" of distributed AI, proving efficient enough to run locally on mobile hardware.
- Optimization Gains: Specific systems have demonstrated massive improvements. ScaleLLM achieved a 4.3x speedup over standard inference engines, while FedKSeed reduced communication costs to under 18 KB per round for fine-tuning billion-parameter models.
Future Directions and Challenges
Despite rapid progress, several challenges remain. Processing long-video sequences in a distributed manner is still computationally intensive. Federated learning continues to suffer from the "straggler problem," where the slowest device bottlenecks the entire network. Furthermore, the total energy footprint of millions of edge devices—the "Green AI" factor—remains poorly understood. Future research is pointing toward Large Concept Models (LCMs) that reason over high-level concepts rather than individual tokens, potentially revolutionizing the efficiency of distributed training.
Summary updated