CECAR: Cache & Expert Co-Aware Routing Accelerates On-Device Inference of MoE LLMs

Byeongju Kim, Hoonki Lee, Byungjun Kim, Inyul Ra, Yongjin Kim, Yongseong Lee, Sangyeob KimarXiv 2026 · arXiv

TL;DR

CECAR (Cache & Expert Co-Aware Routing) is a new inference system that allows large Mixture-of-Experts (MoE) models to run at interactive speeds on consumer-grade hardware. By co-optimizing how a model chooses its "experts" with how it manages its memory cache, CECAR achieves a 4.5x speedup, enabling a 30B-parameter model to run at 12.5 tokens/s on a single 16GB GPU.

What Problem Does This Paper Address?

Large Language Models (LLMs) are getting bigger, but our personal devices aren't. Mixture-of-Experts (MoE) architectures—like those used in Mixtral or DeepSeek—are a popular solution because they are "sparse": for every word generated, the model only activates a small fraction of its total parameters (the "experts").

However, there is a catch. To run these models efficiently, systems usually require all experts to be resident in high-speed memory (VRAM). A 30B-parameter model requires roughly 60GB of memory, far exceeding the 8–16GB found on most consumer GPUs. When experts don't fit in VRAM, they must be fetched from slower storage like an SSD. This I/O bottleneck is so severe that even "optimal" caching strategies result in sluggish performance (often <3 tokens/s), making large MoE models impractical for on-device use.

Methods

The authors propose CECAR, which moves away from the idea that the "router" (the part of the model that picks experts) and the "cache" (the part that stores them) should operate independently. It consists of three main pillars:

  1. Cache-Aware Routing: Instead of picking experts solely based on the model's mathematical preference, CECAR gives a "bonus" to experts already sitting in VRAM. It also uses "cache-conditioned pruning" to skip low-priority experts that aren't currently cached, avoiding expensive SSD fetches.
  2. Expert-Aware Routing: To ensure the model doesn't lose its "intelligence" by picking irrelevant experts just because they are in memory, CECAR uses margin-based adjustments. It only reorders experts within the top-ranked candidates, preserving the semantic intent of the original model.
  3. ML-Based Cache Policy: Rather than using standard rules like LRU (Least Recently Used), CECAR uses a lightweight neural network to predict the "reuse distance" of each expert. It specifically employs two detectors—FFN-Hot for short-term reuse and FFN-Cold for long-term eviction—allowing the system to adapt to different cache sizes and workloads.

Datasets

The authors evaluated CECAR across several diverse benchmarks to ensure both speed and "brain power" were maintained:

  • Models: Qwen3-30B-A3B, DeepSeek-V2-Lite-Chat, and OLMoE-1B-7B.
  • Reasoning & Coding: HumanEval, MBPP (coding), GPQA-Diamond (science), and MATH-500.
  • Cache Training: The ML-based cache policy was trained using routing histories from the TriviaQA dataset.

Key Insights and Ideas

  • The "Belady" Limit: The authors discovered that even with a perfect "crystal ball" (Belady’s optimal policy), standard MoE routing simply doesn't have enough temporal locality to reach high hit rates. You must change the routing behavior itself to favor the cache if you want high speed.
  • Binary is Brittle: Previous attempts at cache-aware routing used a simple "is it in cache?" binary signal. The authors found this causes "routing instability," where the model gets stuck using the same experts repeatedly (a "stickiness" effect), eventually leading to gibberish or degraded accuracy.
  • SSD vs. DRAM: Surprisingly, CECAR fetching from an SSD into VRAM was 1.8x faster than a baseline that kept the entire model in system DRAM. This highlights that the bottleneck isn't just storage speed, but the efficiency of the transfer path to the GPU.

Key Results

  • Throughput: On a consumer GPU with 16GB VRAM, CECAR ran the Qwen3-30B-A3B model at 12.5 tokens/s.
  • Speedup: This represents a 4.5x speedup over conventional MoE inference and significantly outperformed prior state-of-the-art methods like MoCCE (3.1x faster) and DES (1.7x faster).
  • Accuracy: Despite "messing" with the model's internal routing, CECAR maintained high accuracy. Across 12 model-benchmark pairs, the average accuracy change was only -0.5 points, effectively preserving the model's original performance.

Limitations and Future Work

While CECAR is a major step for on-device AI, the authors note a few areas for improvement:

  • Hardware Variability: The tests were done on high-end desktop PCIe interfaces; performance on mobile SoCs (like those in phones or tablets) with unified memory architectures remains to be seen.
  • Complex Workloads: The system hasn't yet been optimized for "batched" serving (handling multiple users at once) or extremely long-context generation.
  • Future Directions: The authors suggest combining CECAR with weight quantization (compressing the experts themselves) and advanced prefetching techniques to further reduce the memory footprint.

For more details, you can find the code at https://github.com/hoonki2002/CECAR.

Summary updated

Abstract

Despite growing interest in on-device LLM deployment, large-scale Mixture-of-Experts (MoE) models remain impractical on resource-constrained devices, as sparse expert activation still requires all expert weights to be memory-resident. To address this, we identify three key observations for on-device MoE inference: (a) cache hit rates are fundamentally bounded even under Belady's optimal policy; (b) these bounds are insufficient for low-latency on-device inference; and (c) prior cache-aware routing based on binary cache presence can induce routing instability. Based on these insights, we propose CECAR (Cache & Expert Co-Aware Routing), a MoE inference system that jointly considers expert routing and cache management. Unlike prior cache-aware routing methods, CECAR uses an ML-based cache policy to predict the expected reuse distance of each expert and prioritizes routing and execution toward experts with smaller predicted reuse distances. On the Qwen3-30B-A3B model, CECAR achieves 12.5 tokens/s on a consumer-grade GPU with only 16 GB of VRAM, where non-resident experts are fetched from SSD, delivering a 4.5x speedup over conventional MoE inference and exceeding the decode speed of baselines that keep the entire model fully resident in DRAM. Code is available at https://github.com/hoonki2002/CECAR.
BibTeX
@misc{kim2026cecar,
  author = {Byeongju Kim and Hoonki Lee and Byungjun Kim and Inyul Ra and Yongjin Kim and Yongseong Lee and Sangyeob Kim},
  title = {{CECAR: Cache \& Expert Co-Aware Routing Accelerates On-Device Inference of MoE LLMs}},
  year = {2026},
  eprint = {2611.20119},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20119},
}
Figure