TL;DR
This paper introduces a novel CPU-GPU collaborative inference framework designed to run large Mixture-of-Experts (MoE) models on memory-limited consumer hardware. By treating GPU memory as an expert cache and offloading cache misses to highly parallelized CPU cores while asynchronously fetching weights in the background, the framework achieves up to a 4.4× speedup over traditional offloading methods without requiring any model modifications.
What Problem Does This Paper Address?
Mixture-of-Experts (MoE) architectures, like Mixtral 8x7B, have become incredibly popular because they offer the performance of massive dense models at a fraction of the computational cost by only activating a small subset of "experts" for any given token.
However, they pose a massive deployment challenge for consumer-grade hardware. While a model like Mixtral 8x7B only processes a fraction of its parameters per token, it still requires roughly 80GB of memory to hold its weights—far exceeding the 24GB capacity of a consumer flagship GPU like the RTX 4090.
Existing solutions rely on "offloading," where the model weights are stored in CPU RAM and transferred to the GPU on-demand. Unfortunately, the PCIe bottleneck introduces severe communication latency. Even advanced "prefetching" techniques that try to predict and fetch weights ahead of time fail because the time it takes to transfer these massive weights over PCIe is significantly longer than the actual GPU computation time, leading to severe execution stalls.
Methods
The authors propose a cooperative framework that rethinks the role of the CPU from a passive storage dump to an active compute partner:
- GPU Expert Caching: Essential, frequently used components (attention layers, routers, and KV caches) reside permanently in the GPU VRAM. The remaining VRAM is allocated as an $N$-index, $M$-way set-associative cache to store a subset of the MoE expert weights using a Least Recently Used (LRU) eviction policy.
- Asynchronous Cache Miss Handling: When the router selects an expert that is cached on the GPU (a cache hit), computation happens instantly on the GPU. If there is a cache miss, instead of stalling the GPU to wait for a weight transfer, the framework offloads the intermediate activation states to the CPU. The CPU then executes the expert computation using highly optimized multi-threading (via OpenMP).
- Dual-Stream Overlapping: While the CPU handles the computation for the missed expert, the framework utilizes NVIDIA's dual-copy engine via independent CUDA streams to asynchronously fetch the missing expert weights into the GPU cache in the background. This ensures the GPU is prepared for subsequent tokens without blocking current execution.
Datasets
The authors analyzed the routing and expert selection behavior of MoE models using the MMLU (Massive Multitask Language Understanding) dataset. The final framework was evaluated using two state-of-the-art open-source MoE models: Mixtral 8x7B and Phi-3.5-MoE.
Key Insights and Ideas
- The Untapped Potential of the CPU: Traditional offloading pipelines treat the CPU as a bottleneck. However, the authors realized that with sufficient multi-threading (controlled by
OMP_NUM_THREADS), computing an expert layer directly on a modern multi-core CPU can actually be faster than the time it takes to transfer those same weights over a PCIe Gen 4 link to the GPU. - Locality in LLM Routing: Expert selection isn't random; it exhibits strong locality. The authors identified two distinct patterns:
- Consecutive Layers Pattern: Routers direct tokens to the exact same expert ID across adjacent layers roughly 44% of the time.
- Consecutive Tokens Pattern: Within the same layer, there is a 40% to 60% probability that an expert chosen for the current token will be reused by the very next token. This temporal and spatial predictability strongly validates the use of an LRU cache strategy.
Key Results
- Significant Speedups: In single-request scenarios using a 24-core CPU, the framework achieved up to a 4.4× speedup for Mixtral 8x7B (reaching 4.8 tokens/sec) and a 4.3× speedup for Phi-3.5-MoE (reaching 10.4 tokens/sec) compared to standard prefetching methods.
- Outperforming State-of-the-Art: The system outperformed Fiddler (a prominent heterogeneous CPU-GPU framework) by approximately 1.6×. Furthermore, unlike Fiddler, it did not suffer from algorithmic bottlenecks when scaling to models with a high number of experts (like Phi-3.5-MoE's 16 experts per layer).
- Energy Efficiency: Even though active CPU multi-threading draws more instantaneous power, the drastic reduction in total token generation time led to massive energy savings. The framework consumed only 29.9% (Mixtral) and 27.8% (Phi-3.5-MoE) of the energy per token required by traditional prefetching methods.
Limitations and Future Work
- Hardware Dependencies: The framework's efficiency is tightly coupled with CPU capabilities. Performance drops significantly on low-core systems (fewer than 2–4 cores) where CPU computation time can no longer mask or compete with transfer latencies.
- Cache Coverage Constraints: Due to VRAM limits on consumer cards, the expert cache can only cover a fraction of the model's layers (e.g., layers 0 to 13 out of 32 in Mixtral 8x7B). Layers beyond cache coverage must run entirely on the CPU. Future work could explore more dynamic cache allocation strategies across layers to maximize hit rates.
Summary updated