TL;DR
MoEZip is a novel KV cache compression method designed specifically for Mixture-of-Experts (MoE) language models. By identifying and preserving context tokens that are critical for stable "expert routing," MoEZip prevents the performance collapse typically seen in MoE models under aggressive compression, outperforming state-of-the-art baselines by over 20 points on long-context benchmarks at a 10% retention ratio.
What Problem Does This Paper Address?
Large Language Models (LLMs) rely on a "KV cache" to store past keys and values, allowing them to generate text without re-processing the entire prompt at every step. However, this cache grows linearly with sequence length, becoming a massive memory bottleneck for long-context applications like document analysis or coding assistants.
While many methods exist to compress the KV cache for standard "dense" models (by evicting less important tokens), these methods often fail when applied to Mixture-of-Experts (MoE) architectures. In an MoE model, a "router" decides which subset of parameters (experts) to activate for each token. The authors discovered that even minor perturbations in the KV cache—caused by token eviction—can cause the router to "flip" and select the wrong experts. This "routing instability" leads to a much sharper decline in generation quality compared to dense models.
Methods
The authors propose MoEZip, a routing-aware compression strategy that prioritizes tokens based on how much they stabilize the model's internal routing decisions.
- Quantifying Routing Fragility: To measure how sensitive a router is to changes, the authors use a metric based on the Fisher information matrix of the routing distribution. They derive a "Fisher-trace metric" that identifies "fragile" tokens—those where a tiny change in input could easily change the expert selection.
- Routing-Aware Anchors: Since we don't know the routing of future answer tokens during the initial "prefill" stage, MoEZip uses the instruction tokens as a proxy. It identifies context tokens that are strongly attended to by "fragile" instruction tokens and labels them as anchors.
- Hybrid Scoring: MoEZip doesn't replace existing compression methods; it enhances them. It first reserves a portion of the KV cache budget for these routing-aware anchors and then lets a "base" compressor (like KVZip or SnapKV) fill the rest of the budget using standard importance scores.
- Adaptive Budgeting: The system dynamically adjusts how many anchors to keep based on the overall fragility of the specific prompt, ensuring that more difficult or "unstable" inputs receive more protection.
Datasets
The researchers evaluated MoEZip across several rigorous long-context benchmarks:
- LongBench: A comprehensive suite of 12 tasks covering QA, summarization, and code completion.
- RULER: A benchmark designed to test the "effective" context window of models across 13 tasks.
- Needle In A Haystack (NIAH): A standard test for information retrieval from long contexts.
The primary models used for testing were Qwen3-30B-A3B (which uses softmax routing) and GLM-4.7-Flash (which uses sigmoid routing), demonstrating the method's versatility across different MoE designs.
Key Insights and Ideas
- The Instruction Proxy: A key "aha" moment was realizing that the routing stability of instruction tokens (which we can see during prefill) is a highly effective predictor of the routing stability for the yet-to-be-generated answer tokens.
- Stability vs. Saliency: Traditional compression focuses on "saliency" (which tokens the model looks at most). MoEZip introduces "stability" as a second, equally important dimension for MoE models. Some tokens might not have the highest attention weight, but they are essential for keeping the model's "brain" (the router) on the right track.
- Compatibility: MoEZip is designed as a modular "plug-in." It can be layered on top of almost any existing dense-model KV compression algorithm, making it easy to integrate into existing inference pipelines.
Key Results
MoEZip shows its greatest strength under aggressive compression (low retention ratios).
- Performance Gains: At a 10% KV retention ratio, MoEZip outperformed the previous state-of-the-art (KVZip) by +24 points on LongBench and +20 points on RULER. This represents a 2.3× and 4.9× improvement in scores, respectively.
- Robustness: While other methods saw performance "collapse" (scores dropping toward zero) as the cache was shrunk, MoEZip maintained remarkably stable performance.
- Generalization: The method proved effective for both softmax and sigmoid-based routers and scaled successfully from 30B to 235B parameter models.
Limitations and Future Work
- White-Box Requirement: MoEZip requires access to the model's internal router outputs and attention statistics. This makes it difficult to use with "black-box" model APIs (like OpenAI's) where only text outputs are available.
- Local Proxy: The Fisher trace is a "local" sensitivity measure. While it works well in practice, it doesn't perfectly account for how errors might compound globally across many layers of a very deep model.
- Future Directions: The authors suggest exploring how routing-aware compression could be applied to multimodal MoE models (handling images and video) or combined with more advanced quantization techniques.
Summary updated