DynaMo: Runtime Switchable Quantization for MoE with Cross-Dataset Adaptation

Zihao Zheng, Xiuping Cui, Size Zheng, Maoliang Li, Jiayu Chen, Yun Liang, Xiang Chen

Abstract

As the Mix-of-Experts (MoE) architecture increases the number of parameters in large models, there is an even greater need for model quantization. However, existing quantization methods overlook the expert dynamics of MoE across multiple datasets. Moreover, the existing static quantization cannot adapt MoE to various data change scenarios. In this paper, we perform a multi-level analysis to reveal MoE dynamics and define the significance of each channel/each expert. Based on the analysis results, we propose DynaMo, an end-to-end MoE quantization framework. DynaMo adopts an expert-level mixed-precision baseline quantization strategy, which ensures the quantized MoEs are compatible with multiple existing datasets. Furthermore, DynaMo incorporates a channel-level dynamic switching mechanism to adapt these quantized MoE models to novel datasets. Experiments show that DynaMo achieves a 2.78~4.54 PPL decrease and a 1.85%~3.77% accuracy improvement in various datasets, with ~3x inference speedup and negligible overhead.

Introduction

Mixture-of-Experts (MoE) architectures, such as Mixtral and DeepSeek, have become prominent in the Large Language Model (LLM) landscape due to their ability to provide massive parameter counts with sparse, efficient activation. However, their sheer size necessitates quantization—compressing weights from 16-bit floats to low-bit integers—for practical deployment.

Existing quantization methods like GPTQ or AWQ are primarily static; they calibrate a model on a single dataset and fix those settings. This approach is ill-suited for MoEs, which are inherently dynamic. Different "experts" are activated depending on the input data distribution. A static quantization scheme calibrated on one task often leads to significant accuracy degradation when the model encounters a different dataset, as it fails to account for shifting "expert dynamics."

The DynaMo Framework

DynaMo is an end-to-end framework designed to treat MoE quantization as a multi-level optimization problem, allowing for runtime adaptation to diverse datasets.

Overview of the DynaMo framework compared to conventional quantization.

1. Multi-Level Analysis

The researchers first analyzed individual channel and expert behaviors across different datasets. They discovered that MoE models exhibit a "one-to-many" mapping, where one token embedding affects multiple experts, and that expert significance shifts significantly between tasks. This analysis forms the basis for identifying which components of the model require higher precision.

2. Expert-Level Mixed-Precision Baseline (MoQ-B)

Instead of calibrating on a single dataset, DynaMo utilizes a "Synthesized Expert Baseline." It clusters experts into four precision tiers (INT2, INT4, INT6, and INT8) based on their importance across a joint distribution of multiple datasets. This ensures a robust baseline that performs well across a variety of known tasks.

The Expert-Level Baseline Quantization (MoQ-B) process.

3. Channel-Level Dynamic Switching

The core innovation of DynaMo is the discovery of the "1% Salience Rule." Analysis revealed that only about 1% of channels in each expert are responsible for the most prominent dynamic shifts across datasets. DynaMo caches these salient channels in their original FP16 format.

4. Runtime Adaptation

When the input data distribution changes, DynaMo avoids full re-quantization. Instead, it performs a "Channel-Level Dynamic Switch," swapping the cached FP16 channels and adjusting their precision tiers to match the new data's requirements. This process is extremely efficient, occurring in milliseconds at runtime.

Experimental Results

The framework was evaluated on several MoE models, including OLMoE (7B), MoE-Girl (7B), Qwen1.5-MoE (14B), and DeepSeek-MoE (16B), across language modeling (WikiText2, C4) and zero-shot benchmarks (ARC, RTE, PIQA, etc.).

  • Accuracy Improvement: DynaMo consistently outperformed static baselines like GPTQ and MoEPTQ, showing a 2.78 to 4.54 point decrease in Perplexity (PPL) and a 1.85% to 3.77% improvement in zero-shot accuracy.
  • Inference Speedup: By implementing custom GPU kernels that pack low-precision weights and use fused dequantization-and-multiply operations, DynaMo achieves a ~3x speedup compared to uncompressed FP16 models.
  • Minimal Overhead: The channel cache for a 7B model is approximately 152MB (~1% of the memory footprint). Switching quantization states takes only 13ms to 21ms, making it practically unnoticeable during inference.

Conclusion and Future Work

DynaMo demonstrates that MoE quantization can be as dynamic as the models themselves. By focusing on the most salient 1% of channels, it achieves high accuracy and significant speedups with minimal memory overhead. Future research may explore automated systems for detecting distribution shifts to trigger switching automatically and extending the framework to include activation quantization (e.g., W4A4).

Summary updated

Abstract

As the Mix-of-Experts (MoE) architecture increases the number of parameters in large models, there is an even greater need for model quantization. However, existing quantization methods overlook the expert dynamics of MoE across multiple datasets. Moreover, the existing static quantization cannot adapt MoE to various data change scenarios. In this paper, we perform a multi-level analysis to reveal MoE dynamics and define the significance of each channel/each expert. Based on the analysis results, we propose \textit{DynaMo}, an end-to-end MoE quantization framework. DynaMo adopts an expert-level mixed-precision baseline quantization strategy, which ensures the quantized MoEs are compatible with multiple existing datasets. Furthermore, DynaMo incorporates a channel-level dynamic switching mechanism to adapt these quantized MoE models to novel datasets. Experiments show that DynaMo achieves a 2.78~4.54 PPL decrease and a 1.85%~3.77% accuracy improvement in various datasets, with ~3x inference speedup and negligible overhead.
BibTeX
@inproceedings{zheng2026dynamo,
  author = {Zihao Zheng and Xiuping Cui and Size Zheng and Maoliang Li and Jiayu Chen and Yun Liang and Xiang Chen},
  title = {{DynaMo: Runtime Switchable Quantization for MoE with Cross-Dataset Adaptation}},
  booktitle = {DATE},
  year = {2026},
  doi = {10.23919/date69613.2026.11539395},
}
Figure