Introduction
Running state-of-the-art Large Language Models (LLMs) on mobile devices presents a significant challenge. While cloud-based LLMs offer high performance, they raise privacy concerns and suffer from network latency. Conversely, on-device LLMs are constrained by hardware limitations, often failing to run large-scale models (e.g., 70B+ parameters) efficiently. WDMoE (Wireless Distributed Mixture of Experts) addresses this by distributing LLM components across a base station and multiple mobile devices within a wireless edge network.
System Architecture
The core of WDMoE lies in leveraging the Mixture of Experts (MoE) architecture. In an MoE model, a gating network directs specific data tokens to a subset of specialized "experts" rather than processing them through a single massive layer.
As shown in the figure above, the system is partitioned:
- Base Station (BS): Hosts the "brain" of the model, including the gating network and heavy attention layers. The BS possesses superior compute power and maintains a global view of the inference task.
- Mobile Devices: Host the individual "experts." Since experts can operate in parallel, they are well-suited for distributed deployment across multiple user devices.
Training-Free Expert Selection
A primary challenge in wireless distributed systems is the instability of connections. To prevent "straggler" devices from bottlenecking inference, WDMoE introduces a training-free selection policy based on the Weight-to-Latency Ratio (WLR):
$$\text{WLR} = \frac{\text{Model Weight}}{\text{Estimated Latency}}$$
This metric balances the importance of an expert's contribution (weight) against the real-time wireless latency of the device hosting it. If a device's signal degrades, its WLR drops, prompting the system to dynamically skip that expert or select a faster alternative.
Performance and Results
The framework was evaluated using the Mixtral 8x7B model across various benchmarks, including MMLU, PIQA, ARC, and HumanEval. Key findings include:
- Accuracy: WDMoE outperformed Llama 2 70B on most benchmarks while activating only ~13B parameters.
- Latency Reduction: On the GSM-8K math benchmark, WDMoE achieved a 30.21% reduction in end-to-end latency compared to standard distributed methods.
- Efficiency: The system demonstrated a 1.65x speedup over the baseline Mixtral model on the PIQA benchmark while maintaining high accuracy.
Conclusion
WDMoE demonstrates that LLMs are naturally robust to distributed expert loss. By treating latency as a primary factor in expert selection, the framework enables high-performance LLM inference on mobile networks without the need for model retraining. Future work will explore incentive models for device participation and more complex splitting strategies to further optimize data transmission.
Summary updated