WDMoE: Wireless Distributed Large Language Models with Mixture of Experts

Nan Xue, Yaping Sun, Zhiyong Chen, Meixia Tao, Xiaodong Xu, Liang Qian, Shuguang Cui, Ping Zhang

Introduction

Running state-of-the-art Large Language Models (LLMs) on mobile devices presents a significant challenge. While cloud-based LLMs offer high performance, they raise privacy concerns and suffer from network latency. Conversely, on-device LLMs are constrained by hardware limitations, often failing to run large-scale models (e.g., 70B+ parameters) efficiently. WDMoE (Wireless Distributed Mixture of Experts) addresses this by distributing LLM components across a base station and multiple mobile devices within a wireless edge network.

System Architecture

The core of WDMoE lies in leveraging the Mixture of Experts (MoE) architecture. In an MoE model, a gating network directs specific data tokens to a subset of specialized "experts" rather than processing them through a single massive layer.

Architecture of the WDMoE framework showing the split between the Base Station and mobile devices.

As shown in the figure above, the system is partitioned:

  • Base Station (BS): Hosts the "brain" of the model, including the gating network and heavy attention layers. The BS possesses superior compute power and maintains a global view of the inference task.
  • Mobile Devices: Host the individual "experts." Since experts can operate in parallel, they are well-suited for distributed deployment across multiple user devices.

Training-Free Expert Selection

A primary challenge in wireless distributed systems is the instability of connections. To prevent "straggler" devices from bottlenecking inference, WDMoE introduces a training-free selection policy based on the Weight-to-Latency Ratio (WLR):

$$\text{WLR} = \frac{\text{Model Weight}}{\text{Estimated Latency}}$$

This metric balances the importance of an expert's contribution (weight) against the real-time wireless latency of the device hosting it. If a device's signal degrades, its WLR drops, prompting the system to dynamically skip that expert or select a faster alternative.

Performance and Results

The framework was evaluated using the Mixtral 8x7B model across various benchmarks, including MMLU, PIQA, ARC, and HumanEval. Key findings include:

  • Accuracy: WDMoE outperformed Llama 2 70B on most benchmarks while activating only ~13B parameters.
  • Latency Reduction: On the GSM-8K math benchmark, WDMoE achieved a 30.21% reduction in end-to-end latency compared to standard distributed methods.
  • Efficiency: The system demonstrated a 1.65x speedup over the baseline Mixtral model on the PIQA benchmark while maintaining high accuracy.

Conclusion

WDMoE demonstrates that LLMs are naturally robust to distributed expert loss. By treating latency as a primary factor in expert selection, the framework enables high-performance LLM inference on mobile networks without the need for model retraining. Future work will explore incentive models for device participation and more complex splitting strategies to further optimize data transmission.

Summary updated

BibTeX
@inproceedings{xue2024wdmoe,
  author = {Nan Xue and Yaping Sun and Zhiyong Chen and Meixia Tao and Xiaodong Xu and Liang Qian and Shuguang Cui and Ping Zhang},
  title = {{WDMoE: Wireless Distributed Large Language Models with Mixture of Experts}},
  booktitle = {GLOBECOM},
  year = {2024},
  doi = {10.1109/globecom52923.2024.10901084},
}
Figure