MoQa: Rethinking MoE Quantization with Multi-stage Data-model Distribution Awareness

Zihao Zheng, Xiuping Cui, Size Zheng, Maoliang Li, Jiayu Chen, Yun Eric Liang, Xiang ChenarXiv 2025 · arXiv · OpenReview

Abstract

With the advances in artificial intelligence, Mix-of-Experts (MoE) has become the main form of Large Language Models (LLMs), and its demand for model compression is increasing. Quantization is an effective method that not only compresses the models but also significantly accelerates their performance. Existing quantization methods have gradually shifted the focus from parameter scaling to the analysis of data distributions. However, their analysis is designed for dense LLMs and relies on the simple one-model-all-data mapping, which is unsuitable for MoEs. This paper proposes a new quantization framework called MoQa. MoQa decouples the data-model distribution complexity of MoEs in multiple analysis stages, quantitively revealing the dynamics during sparse data activation, data-parameter mapping, and inter-expert correlations. Based on these, MoQa identifies particular experts' and parameters' significance with optimal data-model distribution awareness and proposes a series of fine-grained mix-quantization strategies adaptive to various data activation and expert combination scenarios. Moreover, MoQa discusses the limitations of existing quantization and analyzes the impact of each stage analysis, showing novel insights for MoE quantization. Experiments show that MoQa achieves a 1.69~2.18 perplexity decrease in language modeling tasks and a 1.58%~8.91% accuracy improvement in zero-shot inference tasks.

Introduction

Quantization—reducing the precision of model weights from 16-bit floats to low-bit integers—is essential for deploying massive models on consumer hardware. However, standard quantization methods designed for dense architectures often fail when applied to Mixture-of-Experts (MoE) models.

MoE architectures introduce complex one-to-many token-to-expert mappings and dataset-specific dynamics that traditional quantization fails to address.

MoE models present two unique challenges that traditional methods ignore:

  1. One-to-Many Mapping: In dense models, one token maps to one set of weights. In MoEs, a single token is dispatched to multiple experts, creating complex correlations.
  2. Expert Dynamics: MoEs are designed for versatility; different experts activate depending on the input data. Static quantization calibrated on a single dataset becomes a bottleneck when the model encounters new data distributions, leading to significant accuracy drops.

The MoQa Framework

MoQa addresses these challenges by decoupling the data-model distribution complexity through a multi-stage analysis, moving away from "one-size-fits-all" quantization.

Multi-Level Analysis

Before quantization, MoQa performs a deep analysis of expert and channel behavior across diverse datasets. It introduces a "Synthesized Expert Baseline Significance" metric to distinguish between universally critical experts and those specialized for specific tasks.

Expert-Level Baseline Quantization (MoQ-B)

MoQa avoids uniform bit-widths. Instead, it uses a clustering algorithm (Fuzzy C-Means) to group experts into four precision tiers: INT2, INT4, INT6, and INT8. Experts consistently identified as critical across datasets receive higher precision, while less active experts are compressed more aggressively to maximize efficiency.

MoQa employs expert-level mixed-precision by clustering experts into precision tiers based on their cross-dataset significance.

Channel-Level Dynamic Switching (MoQ-D)

To handle runtime distribution shifts, MoQa identifies the top ~1% of channels in each expert that are most sensitive to dataset changes. It caches the original high-precision (FP16) values for these salient channels. At runtime, if a distribution shift is detected, MoQa can instantly adjust the precision of these specific channels, providing the benefits of high-precision adaptation with negligible overhead.

Experimental Results

MoQa was evaluated across several MoE models, including OLMoE, MoE-Girl, Qwen1.5-MoE, and DeepSeek-MoE, using language modeling (WikiText2, C4) and zero-shot tasks (ARC, RTE, PIQA, etc.).

  • Accuracy and Perplexity: MoQa consistently outperformed state-of-the-art baselines like GPTQ, improving zero-shot accuracy by 1.85% to 3.77% and reducing perplexity by 2.78 to 4.54 points.
  • Inference Speed: The framework achieved a 2.91× to 3.08× speedup compared to unquantized FP16 baselines.
  • Efficiency: The dynamic switching mechanism adds less than 11% overhead to online inference, and the channel cache occupies only ~1% of total model memory.

Discussion and Conclusion

A key insight from MoQa is the "1% Rule": targeting only the most salient channels allows for effective dynamic adaptation without the prohibitive costs of full-model dynamic quantization. Furthermore, the framework includes hardware-aware optimizations like quantized weight packing to ensure mixed-precision does not degrade GPU performance. While highly effective, future work could focus on automating runtime distribution shift detection and further optimizing the memory hierarchy for channel caching. MoQa represents a significant step forward in making MoE quantization both smart and adaptable for diverse real-world applications.

Summary updated

Abstract

With the advances in artificial intelligence, Mix-of-Experts (MoE) has become the main form of Large Language Models (LLMs), and its demand for model compression is increasing. Quantization is an effective method that not only compresses the models but also significantly accelerates their performance. Existing quantization methods have gradually shifted the focus from parameter scaling to the analysis of data distributions. However, their analysis is designed for dense LLMs and relies on the simple one-model-all-data mapping, which is unsuitable for MoEs. This paper proposes a new quantization framework called MoQa. MoQa decouples the data-model distribution complexity of MoEs in multiple analysis stages, quantitively revealing the dynamics during sparse data activation, data-parameter mapping, and inter-expert correlations. Based on these, MoQa identifies particular experts' and parameters' significance with optimal data-model distribution awareness and proposes a series of fine-grained mix-quantization strategies adaptive to various data activation and expert combination scenarios. Moreover, MoQa discusses the limitations of existing quantization and analyzes the impact of each stage analysis, showing novel insights for MoE quantization. Experiments show that MoQa achieves a 1.69~2.18 perplexity decrease in language modeling tasks and a 1.58%~8.91% accuracy improvement in zero-shot inference tasks. We believe MoQa will play a role in future MoE construction, optimization, and compression.
BibTeX
@misc{zheng2025moqa,
  author = {Zihao Zheng and Xiuping Cui and Size Zheng and Maoliang Li and Jiayu Chen and Yun Eric Liang and Xiang Chen},
  title = {{MoQa: Rethinking MoE Quantization with Multi-stage Data-model Distribution Awareness}},
  year = {2025},
  eprint = {2503.21135},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.21135},
}
Figure