Efficient CPU-GPU Collaborative Inference for MoE-based LLMs on Memory-Limited Systems

En-Ming Huang, Li-Shang Lin, Chun-Yi Lee

TL;DR

This paper introduces a novel CPU-GPU collaborative inference framework designed to run large Mixture-of-Experts (MoE) models on memory-limited consumer hardware. By treating GPU memory as an expert cache and offloading cache misses to highly parallelized CPU cores while asynchronously fetching weights in the background, the framework achieves up to a 4.4× speedup over traditional offloading methods without requiring any model modifications.

What Problem Does This Paper Address?

Mixture-of-Experts (MoE) architectures, like Mixtral 8x7B, have become incredibly popular because they offer the performance of massive dense models at a fraction of the computational cost by only activating a small subset of "experts" for any given token.

Fig. 1: Comparison of dense FFN and MoE layer architectures.

However, they pose a massive deployment challenge for consumer-grade hardware. While a model like Mixtral 8x7B only processes a fraction of its parameters per token, it still requires roughly 80GB of memory to hold its weights—far exceeding the 24GB capacity of a consumer flagship GPU like the RTX 4090.

Existing solutions rely on "offloading," where the model weights are stored in CPU RAM and transferred to the GPU on-demand. Unfortunately, the PCIe bottleneck introduces severe communication latency. Even advanced "prefetching" techniques that try to predict and fetch weights ahead of time fail because the time it takes to transfer these massive weights over PCIe is significantly longer than the actual GPU computation time, leading to severe execution stalls.

Methods

The authors propose a cooperative framework that rethinks the role of the CPU from a passive storage dump to an active compute partner:

  1. GPU Expert Caching: Essential, frequently used components (attention layers, routers, and KV caches) reside permanently in the GPU VRAM. The remaining VRAM is allocated as an $N$-index, $M$-way set-associative cache to store a subset of the MoE expert weights using a Least Recently Used (LRU) eviction policy.
  2. Asynchronous Cache Miss Handling: When the router selects an expert that is cached on the GPU (a cache hit), computation happens instantly on the GPU. If there is a cache miss, instead of stalling the GPU to wait for a weight transfer, the framework offloads the intermediate activation states to the CPU. The CPU then executes the expert computation using highly optimized multi-threading (via OpenMP).
  3. Dual-Stream Overlapping: While the CPU handles the computation for the missed expert, the framework utilizes NVIDIA's dual-copy engine via independent CUDA streams to asynchronously fetch the missing expert weights into the GPU cache in the background. This ensures the GPU is prepared for subsequent tokens without blocking current execution.
Workflow overview and timeline comparison.

Datasets

The authors analyzed the routing and expert selection behavior of MoE models using the MMLU (Massive Multitask Language Understanding) dataset. The final framework was evaluated using two state-of-the-art open-source MoE models: Mixtral 8x7B and Phi-3.5-MoE.

Key Insights and Ideas

  • The Untapped Potential of the CPU: Traditional offloading pipelines treat the CPU as a bottleneck. However, the authors realized that with sufficient multi-threading (controlled by OMP_NUM_THREADS), computing an expert layer directly on a modern multi-core CPU can actually be faster than the time it takes to transfer those same weights over a PCIe Gen 4 link to the GPU.
  • Locality in LLM Routing: Expert selection isn't random; it exhibits strong locality. The authors identified two distinct patterns:
  • Consecutive Layers Pattern: Routers direct tokens to the exact same expert ID across adjacent layers roughly 44% of the time.
  • Consecutive Tokens Pattern: Within the same layer, there is a 40% to 60% probability that an expert chosen for the current token will be reused by the very next token. This temporal and spatial predictability strongly validates the use of an LRU cache strategy.
Fig. 2: Mixtral 8x7B (32 Transformer blocks) expert selection patterns on the MMLU [22], [23] dataset: (1) Consecutive Layers Pattern and (2) Consecutive Token Pattern. Each Transformer block’s router demonstrates clear patterns of expert reuse.

Key Results

  • Significant Speedups: In single-request scenarios using a 24-core CPU, the framework achieved up to a 4.4× speedup for Mixtral 8x7B (reaching 4.8 tokens/sec) and a 4.3× speedup for Phi-3.5-MoE (reaching 10.4 tokens/sec) compared to standard prefetching methods.
  • Outperforming State-of-the-Art: The system outperformed Fiddler (a prominent heterogeneous CPU-GPU framework) by approximately 1.6×. Furthermore, unlike Fiddler, it did not suffer from algorithmic bottlenecks when scaling to models with a high number of experts (like Phi-3.5-MoE's 16 experts per layer).
Fig. 3: Mixtral 8x7B token generation speed vs. CPU cores.
  • Energy Efficiency: Even though active CPU multi-threading draws more instantaneous power, the drastic reduction in total token generation time led to massive energy savings. The framework consumed only 29.9% (Mixtral) and 27.8% (Phi-3.5-MoE) of the energy per token required by traditional prefetching methods.

Limitations and Future Work

  • Hardware Dependencies: The framework's efficiency is tightly coupled with CPU capabilities. Performance drops significantly on low-core systems (fewer than 2–4 cores) where CPU computation time can no longer mask or compete with transfer latencies.
  • Cache Coverage Constraints: Due to VRAM limits on consumer cards, the expert cache can only cover a fraction of the model's layers (e.g., layers 0 to 13 out of 32 in Mixtral 8x7B). Layers beyond cache coverage must run entirely on the CPU. Future work could explore more dynamic cache allocation strategies across layers to maximize hit rates.

Summary updated

Abstract

Large Language Models (LLMs) have achieved impressive results across various tasks, yet their high computational demands pose deployment challenges, especially on consumer-grade hardware. Mixture of Experts (MoE) models provide an efficient solution through selective activation of parameter subsets, which reduces computation requirements. Despite this efficiency, state-of-the-art MoE models still require substantial memory beyond typical consumer GPU capacities. Traditional offloading methods that transfer model weights between CPU and GPU introduce latency, limiting inference performance. This paper presents a novel CPU-GPU collaborative inference framework that incorporates an expert caching mechanism on the GPU to reduce data transfer requirements and enable faster inference through cache hits. Computations are offloaded to CPU for efficient cache miss handling, which benefits from CPU multithreading optimizations. The evaluations of our framework demonstrate performance improvements and highlight the potential of CPU-GPU collaboration to maximize hardware utilization for single-request inference scenarios on consumer-grade systems. The implementation of our framework is available at https://github.com/elsa-lab/MoE-CPU-GPU-Collaborative-Inference.
BibTeX
@misc{huang2025efficient,
  author = {En-Ming Huang and Li-Shang Lin and Chun-Yi Lee},
  title = {{Efficient CPU-GPU Collaborative Inference for MoE-based LLMs on Memory-Limited Systems}},
  year = {2025},
  eprint = {2512.16473},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.16473},
}
Figure