Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directions

M. Hadi Amini, Md. Jueal Mia, Yasaman Saadati, Ahmed Imteaj, Seyedsina Nabavirazavi, Urmish Thakker, Md. Zarif Hossain, Awal Ahmed Fime, S. S. IyengararXiv 2025 · OpenReview

Abstract

Language models (LMs) are machine learning models designed to predict linguistic patterns by estimating the probability of word sequences based on large-scale datasets. While larger datasets typically enhance performance, scalability remains a challenge due to constraints in computational power and resources. Distributed computing strategies offer essential solutions for improving scalability and managing growing demand. Further, the use of sensitive datasets raises significant privacy concerns. This paper presents a survey on distributed solutions for various LMs, including large language models (LLMs), vision language models (VLMs), multimodal LLMs (MLLMs), and small language models (SLMs). We review key advancements across the MLLM pipeline, including distributed training, inference, fine-tuning, and deployment, while identifying contributions, limitations, and future areas of improvement.

Introduction: The Centralization Bottleneck

The AI revolution is currently constrained by what can be termed a "centralization tax." State-of-the-art models like GPT-4 or Llama-3 require thousands of high-end GPUs and massive, often sensitive, datasets. This creates two primary problems: scalability and cost, where training models with hundreds of billions of parameters is prohibitively expensive; and privacy, where training on user data in a central cloud poses significant security risks. This survey explores how to distribute the training, fine-tuning, and inference of these models across a network of diverse, potentially low-power devices while keeping data private and local.

Taxonomy of Distributed Strategies

As a comprehensive survey, this work categorizes the current state-of-the-art across six primary focus areas of decentralization:

  • Distributed Training: Utilizing model, data, and pipeline parallelism to split massive workloads.
  • Distributed Inference & Optimization: Implementing systems like FastServe that use preemptive scheduling to reduce latency.
  • Federated Learning (FL) & Fine-tuning: Employing methods like FedIT and FedKSeed to allow models to learn from decentralized data without it ever leaving the user's device.
  • Computing Infrastructures: Leveraging a spectrum of hardware from massive GPU clusters to low-cost edge devices like the NVIDIA Jetson Orin.
  • Communication Efficiency: Reducing the "bandwidth tax" of model updates using quantization and gradient compression.
  • Mobile Edge Intelligence: Bringing LLM capabilities directly to smartphones and IoT sensors.
Figure 2: Distribution of surveyed papers across the six primary focus areas, distinguishing between primary and secondary research focus.

The MLLM Pipeline

A critical contribution of this survey is the detailed analysis of the Multimodal Large Language Model (MLLM) pipeline. Unlike text-only models, MLLMs must handle multiple modalities (text, image, audio) and integrate them into a shared embedding space. This requires unique alignment modules and specialized encoders (e.g., CLIP for images) that have distinct computational profiles, complicating the distribution process.

Figure 4: The comprehensive pipeline for MLLM development, from data collection to deployment, highlighting alignment strategies like RLHF and DPO.

Key Insights and Results

Research indicates a significant shift in the field:

  • Communication is the New Compute: In distributed settings, the time spent moving data between devices often exceeds calculation time. Recent research focuses on "smarter communication," such as hiding latency behind computation.
  • The Rise of SLMs: Small Language Models (1B–7B parameters) are emerging as the "unsung heroes" of distributed AI, proving efficient enough to run locally on mobile hardware.
  • Optimization Gains: Specific systems have demonstrated massive improvements. ScaleLLM achieved a 4.3x speedup over standard inference engines, while FedKSeed reduced communication costs to under 18 KB per round for fine-tuning billion-parameter models.

Future Directions and Challenges

Despite rapid progress, several challenges remain. Processing long-video sequences in a distributed manner is still computationally intensive. Federated learning continues to suffer from the "straggler problem," where the slowest device bottlenecks the entire network. Furthermore, the total energy footprint of millions of edge devices—the "Green AI" factor—remains poorly understood. Future research is pointing toward Large Concept Models (LCMs) that reason over high-level concepts rather than individual tokens, potentially revolutionizing the efficiency of distributed training.

Summary updated

Abstract

Language models (LMs) are machine learning models designed to predict linguistic patterns by estimating the probability of word sequences based on large-scale datasets, such as text. LMs have a wide range of applications in natural language processing (NLP) tasks, including autocomplete and machine translation. Although larger datasets typically enhance LM performance, scalability remains a challenge due to constraints in computational power and resources. Distributed computing strategies offer essential solutions for improving scalability and managing the growing computational demand. Further, the use of sensitive datasets in training and deployment raises significant privacy concerns. Recent research has focused on developing decentralized techniques to enable distributed training and inference while utilizing diverse computational resources and enabling edge AI. This paper presents a survey on distributed solutions for various LMs, including large language models (LLMs), vision language models (VLMs), multimodal LLMs (MLLMs), and small language models (SLMs). While LLMs focus on processing and generating text, MLLMs are designed to handle multiple modalities of data (e.g., text, images, and audio) and to integrate them for broader applications. To this end, this paper reviews key advancements across the MLLM pipeline, including distributed training, inference, fine-tuning, and deployment, while also identifying the contributions, limitations, and future areas of improvement. Further, it categorizes the literature based on six primary focus areas of decentralization. Our analysis describes gaps in current methodologies for enabling distributed solutions for LMs and outline future research directions, emphasizing the need for novel solutions to enhance the robustness and applicability of distributed LMs.
BibTeX
@misc{amini2025distributed,
  author = {M. Hadi Amini and Md. Jueal Mia and Yasaman Saadati and Ahmed Imteaj and Seyedsina Nabavirazavi and Urmish Thakker and Md. Zarif Hossain and Awal Ahmed Fime and S. S. Iyengar},
  title = {{Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directions}},
  year = {2025},
  eprint = {2503.16585},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.16585},
}
Figure