EAT: Expert Account Tracker for Efficient MoE Inference

Yuexian Li, Yifei Yang, zouying cao, hai zhaoSubmission to ICLR 2026 · Rejected

TL;DR

Expert Account Tracker (EAT) is a novel inference strategy that makes Mixture-of-Experts (MoE) models significantly more efficient by tracking the "historical performance" of individual experts. By combining this history-awareness with an adaptive thresholding mechanism, EAT reduces the number of activated experts by over 25% while maintaining—and sometimes exceeding—the accuracy and generation speed of standard methods.

What Problem Does This Paper Address?

Large Language Models (LLMs) often use a Mixture-of-Experts (MoE) architecture to scale up parameters without a linear increase in computational cost. In a standard MoE, a "router" selects a fixed number of specialized sub-networks (experts) to process each token.

However, current MoE inference suffers from two main inefficiencies:

  1. Lack of Flexibility: The "vanilla" strategy usually activates a fixed number of experts (e.g., Top-2), regardless of whether the input token actually needs that much processing power.
  2. Historical Blindness: Existing dynamic selection methods (like Top-P) only look at the current token's routing score. They ignore whether an expert has been consistently reliable or just happens to have a high "noisy" score for a single moment.

This leads to redundant activations, where the model wastes energy on experts that don't contribute much to the final output.

Methods

The authors introduce EAT, which treats expert selection like a managed account. The system consists of three core components:

  • Comprehensive Expert Scoring: Instead of just looking at the current router score, EAT maintains a "history-based importance score" for each expert. This score is calculated using three indicators:
  • Activation Count ($N$): How often the expert is used.
  • Total Probability ($P_{total}$): The cumulative weight assigned to it by the router.
  • Contribution Score ($C_s$): An exponentially smoothed measure of the expert's actual output magnitude (L2 norm).
  • Adaptive Thresholding ($\tau$): Rather than picking a fixed number of experts, EAT calculates a dynamic threshold. This threshold is "performance-aware"—if the model's perplexity (a measure of uncertainty) increases, the threshold drops to allow more experts to help. If the model is confident, the threshold rises to save computation.
  • Dual-Constraint Activation: During inference, EAT selects all experts whose final score exceeds the dynamic threshold, but it caps this at a maximum number ($K$) to ensure the computational budget doesn't explode.

Datasets

The researchers evaluated EAT using the OpenCompass framework, a comprehensive benchmark suite for LLMs. They tested across five dimensions:

  • Reasoning: HellaSwag, PIQA.
  • Language: CHID, WSC.
  • Knowledge: BoolQ.
  • Examination: MMLU, CMMLU.
  • Understanding: XSum.

These tests were performed on two popular MoE models: Mixtral-8x7B-v0.1 and Phi-3.5-MoE-instruct.

Key Insights and Ideas

  • Experts Have "Track Records": The most significant insight is that an expert's past utility is a strong predictor of its current value. By incorporating historical contribution ($C_s$), EAT can distinguish between "consistently reliable" experts and those that are just "noisy."
  • Layer-Wise Sensitivity: Through ablation studies, the authors found that not all layers are created equal. Lower layers (0-23 in Mixtral) are less critical and can handle more aggressive pruning (activating only 1.3 to 1.5 experts). Higher layers (24-30) handle more complex information and require more experts to maintain performance.
  • The "Diminishing Returns" of Sparsity: There is a clear floor for efficiency. Reducing the average activated experts to 1.0 causes a sharp spike in perplexity (error). The authors identified ~1.36 experts per token as a "sweet spot" where performance remains stable but efficiency is maximized.

Key Results

  • Efficiency Gains: EAT achieved an average 25% reduction in the number of activated experts compared to the vanilla Top-2 method. For example, on Mixtral reasoning tasks, it dropped from 2.0 experts to just 1.46.
  • Performance Maintenance: Despite using fewer experts, EAT consistently outperformed the Top-P baseline (a common dynamic selection method) across almost all benchmarks.
  • Speedup: EAT improved token generation speed. In short-sequence settings for the Phi-3.5 model, EAT reached 1.14 tokens/sec, surpassing both the vanilla (0.80) and Top-P (0.87) methods.

Limitations and Future Work

  • Hyperparameter Sensitivity: The method relies on several hyperparameters (like $\alpha, \beta, w_1, w_2, w_3$) to balance history vs. current input. While the authors found stable defaults, these might need tuning for different model architectures.
  • Static vs. Dynamic Layer Routing: While the paper identifies that higher layers are more important, the current EAT implementation applies the same logic across all layers. The authors suggest that a layer-wise routing mechanism—specifically tailored to the needs of lower vs. higher layers—is a promising direction for future research.
  • Training-Free vs. Training-Aware: EAT is a plug-and-play inference strategy. Future work could explore whether integrating "account tracking" directly into the training phase could yield even higher sparsity.

Summary updated

Abstract

Mixture-of-Experts (MoE) models have emerged as a revolutionary method to scale Transformer models. However, traditional MoE architecture still suffers from inefficiency since a large number of experts are unnecessarily activated. Existing approaches for reducing the number of activated experts often overlook the historical performance of each expert. In this paper, we propose EAT, a novel method called $\textbf{Expert Account Tracker (EAT)}$, which utilizes history-awareness metrics and adaptive thresholding to dynamically select the most important experts, thereby reducing the activated expert number while effectively maintaining the model performance. Experiments show that EAT outperforms the existing baseline Top-P method across multiple models and datasets, achieving over 25% an average reduction compared to the vanilla method in the number of activated experts and performing better token generation speed compared to the baseline. Additionally, through ablation studies, we find that excessively reducing the number of activated experts can significantly harm model performance, and the importance of experts varies across layers, with higher-level experts being generally more critical.
BibTeX
@misc{li2026eat,
  author = {Yuexian Li and Yifei Yang and zouying cao and hai zhao},
  title = {{EAT: Expert Account Tracker for Efficient MoE Inference}},
  year = {2026},
  eprint = {2609.33614},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.33614},
}

Loading related content...

Figure