MoEZip: Routing-Aware KV Cache Compression for Sparse Mixture-of-Experts LLMs

Minsang Kim, Seungjun BaekarXiv 2026 · arXiv

TL;DR

MoEZip is a novel KV cache compression method designed specifically for Mixture-of-Experts (MoE) language models. By identifying and preserving context tokens that are critical for stable "expert routing," MoEZip prevents the performance collapse typically seen in MoE models under aggressive compression, outperforming state-of-the-art baselines by over 20 points on long-context benchmarks at a 10% retention ratio.

What Problem Does This Paper Address?

Large Language Models (LLMs) rely on a "KV cache" to store past keys and values, allowing them to generate text without re-processing the entire prompt at every step. However, this cache grows linearly with sequence length, becoming a massive memory bottleneck for long-context applications like document analysis or coding assistants.

While many methods exist to compress the KV cache for standard "dense" models (by evicting less important tokens), these methods often fail when applied to Mixture-of-Experts (MoE) architectures. In an MoE model, a "router" decides which subset of parameters (experts) to activate for each token. The authors discovered that even minor perturbations in the KV cache—caused by token eviction—can cause the router to "flip" and select the wrong experts. This "routing instability" leads to a much sharper decline in generation quality compared to dense models.

Figure 1: Router instability is associated with answer-quality degradation. As KV compression becomes more aggressive, ex- pert selection changes more often, and per- formance drops accordingly.

Methods

The authors propose MoEZip, a routing-aware compression strategy that prioritizes tokens based on how much they stabilize the model's internal routing decisions.

  1. Quantifying Routing Fragility: To measure how sensitive a router is to changes, the authors use a metric based on the Fisher information matrix of the routing distribution. They derive a "Fisher-trace metric" that identifies "fragile" tokens—those where a tiny change in input could easily change the expert selection.
  2. Routing-Aware Anchors: Since we don't know the routing of future answer tokens during the initial "prefill" stage, MoEZip uses the instruction tokens as a proxy. It identifies context tokens that are strongly attended to by "fragile" instruction tokens and labels them as anchors.
  3. Hybrid Scoring: MoEZip doesn't replace existing compression methods; it enhances them. It first reserves a portion of the KV cache budget for these routing-aware anchors and then lets a "base" compressor (like KVZip or SnapKV) fill the rest of the budget using standard importance scores.
  4. Adaptive Budgeting: The system dynamically adjusts how many anchors to keep based on the overall fragility of the specific prompt, ensuring that more difficult or "unstable" inputs receive more protection.
Figure 2: Overview of MoEZip. MoEZip estimates routing fragility from instruction-token router states, transfers it to context positions via instruction-to-context attention, and reserves the top-scoring positions as routing-aware anchors. The base scorer fills the remaining budget, and thus the final cache size and decoding procedure remain unchanged.

Datasets

The researchers evaluated MoEZip across several rigorous long-context benchmarks:

  • LongBench: A comprehensive suite of 12 tasks covering QA, summarization, and code completion.
  • RULER: A benchmark designed to test the "effective" context window of models across 13 tasks.
  • Needle In A Haystack (NIAH): A standard test for information retrieval from long contexts.

The primary models used for testing were Qwen3-30B-A3B (which uses softmax routing) and GLM-4.7-Flash (which uses sigmoid routing), demonstrating the method's versatility across different MoE designs.

Key Insights and Ideas

  • The Instruction Proxy: A key "aha" moment was realizing that the routing stability of instruction tokens (which we can see during prefill) is a highly effective predictor of the routing stability for the yet-to-be-generated answer tokens.
  • Stability vs. Saliency: Traditional compression focuses on "saliency" (which tokens the model looks at most). MoEZip introduces "stability" as a second, equally important dimension for MoE models. Some tokens might not have the highest attention weight, but they are essential for keeping the model's "brain" (the router) on the right track.
  • Compatibility: MoEZip is designed as a modular "plug-in." It can be layered on top of almost any existing dense-model KV compression algorithm, making it easy to integrate into existing inference pipelines.

Key Results

MoEZip shows its greatest strength under aggressive compression (low retention ratios).

  • Performance Gains: At a 10% KV retention ratio, MoEZip outperformed the previous state-of-the-art (KVZip) by +24 points on LongBench and +20 points on RULER. This represents a 2.3× and 4.9× improvement in scores, respectively.
  • Robustness: While other methods saw performance "collapse" (scores dropping toward zero) as the cache was shrunk, MoEZip maintained remarkably stable performance.
  • Generalization: The method proved effective for both softmax and sigmoid-based routers and scaled successfully from 30B to 235B parameter models.
Figure 3: Performance on 12 LongBench tasks across different KV cache ratios. MoEZip consistently preserves higher task performance than existing KV compression baselines, especially under aggressive compression.

Limitations and Future Work

  • White-Box Requirement: MoEZip requires access to the model's internal router outputs and attention statistics. This makes it difficult to use with "black-box" model APIs (like OpenAI's) where only text outputs are available.
  • Local Proxy: The Fisher trace is a "local" sensitivity measure. While it works well in practice, it doesn't perfectly account for how errors might compound globally across many layers of a very deep model.
  • Future Directions: The authors suggest exploring how routing-aware compression could be applied to multimodal MoE models (handling images and video) or combined with more advanced quantization techniques.

Summary updated

Abstract

Mixture-of-Experts (MoE) language models scale model capacity efficiently by activating a small subset of parameters per token. However, in long-context inference, memory remains a major bottleneck because the KV cache grows linearly with sequence length. Existing KV cache compression methods reduce this cost for dense transformers, but overlook a problem specific to MoE: KV eviction can change which experts are selected during answer generation, degrading the generation quality. We propose MoEZip, a routing-aware KV cache compression for sparse MoE language models. To estimate the sensitivity of expert routing to perturbation, we propose a novel metric based on the Fisher information matrix of routing distributions. MoEZip combines this metric with attention weights for routing-aware scoring of contexts. Finally, we compose a complementary scoring system that combines eviction methods for dense and sparse-MoE architectures. Experiments show that MoEZip achieves robust performance under aggressive cache compression on various benchmarks. At a KV retention ratio of 10\%, MoEZip improves over the SoTA baseline by +24 and +20 points on average on LongBench and RULER, corresponding to 2.3$\times$ and 4.9$\times$ higher scores, respectively.
BibTeX
@misc{kim2026moezip,
  author = {Minsang Kim and Seungjun Baek},
  title = {{MoEZip: Routing-Aware KV Cache Compression for Sparse Mixture-of-Experts LLMs}},
  year = {2026},
  eprint = {2610.30206},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.30206},
}
Figure