Memory Efficient Inference for Mixture of Experts Models

763 concepts505 papers4 proposals

Mixture-of-Experts (MoE) architectures present a compelling paradox in modern machine learning. By routing each input token to only a small subset of specialized "expert" sub-networks, MoEs decouple a model's total parameter capacity from its per-token computational cost. In theory, this allows a trillion-parameter model to execute with the speed and FLOP budget of a much smaller dense model.

In practice, however, deployment faces a severe hardware bottleneck: the static memory overhead. While only a fraction of the experts are active at any single compute step, the entire pool of experts must typically reside in high-speed memory (VRAM) to avoid catastrophic latency penalties during routing. When a model's total size scales to tens or hundreds of gigabytes, it quickly exceeds the memory capacity of consumer-grade or single-GPU hardware. Furthermore, as Ruisi Cai et al. observe, when processing batches of diverse tokens, the fragmentation of expert requests often forces the system to load nearly all experts simultaneously anyway, undermining the theoretical memory savings of sparse activation.

To transition MoEs from massive data centers to resource-constrained edge devices and single-GPU workstations, researchers have developed a multi-faceted optimization paradigm. This direction focuses on shrinking the static memory footprint of experts, orchestrating smart data transfers between system memory and hardware accelerators, and modifying routing algorithms to respect hardware boundaries.

Pruning and Skipping Redundant Experts

The most direct way to alleviate memory pressure is to reduce the total number of experts permanently or dynamically. Although MoE models are trained to distribute knowledge across many experts, not all experts contribute equally to every task.

In multilingual settings, for instance, experts naturally specialize in specific languages or language families. Yeskendir Koishekenov et al. demonstrated that up to 80% of the experts in a massively multilingual model like NLLB-200 can be safely pruned post-training without requiring any retraining. By identifying and retaining only the language-specific experts required for a target translation pair, the authors reduced the deployment requirement from four high-end GPUs down to a single device.

For general-purpose large language models, pruning can be guided by activation frequencies or differentiable metrics. Xudong Lu et al. introduced post-training techniques that combine permanent Expert Pruning with Dynamic Expert Skipping. By permanently removing underutilized experts and temporarily bypassing others during inference when their routing weights fall below a threshold, they halved the VRAM requirements for models like Mixtral 8x7B. To make this selection more optimal, Sikai Bai et al. formulated the reduction as a differentiable optimization problem (DiEP), automatically learning which layers and specific experts are critical to preserve performance while aggressively cutting the overall expert count.

[Standard MoE Layer]  --> All experts must reside in VRAM (Massive Footprint)
       │
       ├──> [Expert Pruning]   --> Permanently delete underutilized experts
       └──> [Expert Skipping]  --> Dynamically bypass experts below activation threshold

Hardware-Aware and Residency-Aware Routing

When the model footprint still exceeds VRAM, parts of the model must be offloaded to slower secondary storage (such as system DRAM or SSDs). The core challenge then shifts from minimizing capacity to minimizing the latency of fetching weights over slow hardware buses (like PCIe).

Instead of treating the router as a purely mathematical function maximizing token-expert affinity, recent work infuses the router with hardware awareness. The intuition is simple: if an expert is already loaded in fast memory, prefer using it.

Token Input ──> [Residency-Aware Router]
                     │
                     ├──> Expert A (In HBM/VRAM)  ──> High Priority (Fast Execution)
                     └──> Expert B (In DRAM/SSD)  ──> Low Priority (Avoids PCIe Bottleneck)

Abhimanyu Rajeshkumar Bambhaniya et al. developed MoE-ERAS (Expert Residency Aware Selection), a strategy that biases the routing mechanism toward experts currently residing in high-bandwidth memory (HBM). Because accessing an on-chip expert is orders of magnitude faster than fetching one from off-chip memory, this bias yields significant latency reductions with minimal impact on accuracy. Similarly, Andrii Skliar et al. applied this concept to mobile devices, biasing the model to reuse experts already cached in DRAM to prevent the memory thrashing that typically cripples real-time generation on edge hardware.

Complementing these algorithmic changes are system-level orchestration frameworks. Keisuke Kamahori et al. introduced Fiddler, a CPU-GPU orchestration system that dynamically decides whether to copy expert weights to the GPU or offload the data activations to the CPU based on the current workload. This bidirectional scheduling helps hide the latency of weight transfers behind active computation.

Parameter-Efficient and Low-Bitwidth Experts

Another highly effective strategy focuses on reducing the size of individual experts rather than their count. When dense models are "upcycled" into MoEs, initializing each expert as a full copy of a feed-forward network causes an immediate explosion in parameter storage.

To counteract this, Peng Ye et al. proposed DeRS (Decompose, Replace, and Synthesis), which re-frames experts as minor "delta" adjustments to a shared base weight. This approach drastically reduces the unique parameter footprint of each added expert. In a similar vein, Ted Zadouri et al. combined MoE structures with Parameter-Efficient Fine-Tuning (PEFT) by introducing Mixture of LoRA (MoLORA), replacing heavy, full-sized expert layers with lightweight low-rank adapter matrices that are vastly cheaper to store and swap in memory.

Traditional Upcycled MoE:
[Expert 1: Full FFN] [Expert 2: Full FFN] [Expert 3: Full FFN] -> High Memory

Delta / Parameter-Efficient MoE:
       ┌─────────────────── [Shared Base FFN Weight] ───────────────────┐
       │                                │                               │
[Delta Expert 1 (LoRA)]   [Delta Expert 2 (LoRA)]   [Delta Expert 3 (LoRA)] -> Low Memory

Finally, quantization compresses the numerical representation of the expert weights. Hongyu Wang et al. designed MoTE (Mixture of Ternary Experts), which utilizes ultra-low 1.58-bit ternary representations for the routed experts while maintaining a high-precision shared expert to stabilize representation quality. To optimize execution speed alongside storage, Haojie Duanmu et al. developed MxMoE, a co-design framework that applies mixed-precision quantization to individual linear blocks depending on their activation frequency, pairing the compressed weights with custom GPU kernels to maximize throughput.

Exploiting Temporal and Structural Sparsity

Beyond compressing weights or altering routers, memory overhead can be mitigated by restructuring how tokens are processed. Standard MoE models route at the individual token level, meaning that token $t$ and token $t+1$ might require entirely different experts, forcing rapid memory swapping.

Chenyang Song et al. addressed this with BlockFFN, an architecture optimized for chunk-level activation sparsity. By forcing consecutive groups of tokens to share a common set of experts, the system minimizes memory I/O overhead during parallel processing and speculative decoding. This temporal alignment ensures that once an expert's weights are loaded into the processor's cache, they are fully utilized across multiple tokens before being evicted, effectively resolving the memory thrashing problem inherent to token-level routing on edge devices.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 472 papers
  2. Version 1 · 2026-10-04 · 785 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_sparse_model_memory_overhead_a5016161,
  title = {sparse model memory overhead},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sparse-model-memory-overhead/rdr_a5016161a5024e19ac958192e0e6cce3},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_sparse_model_memory_overhead_a5016161_v2,
  title = {sparse model memory overhead},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sparse-model-memory-overhead/rdr_a5016161a5024e19ac958192e0e6cce3/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake