Mixture-of-Experts (MoE) language models have many feed-forward "expert" blocks per layer and send each token to only a few of them. Methods that compress these models by storing weights at lower numerical precision (quantization) decide how many bits each expert gets under one shared assumption: fewer bits means proportionally less memory traffic, and therefore proportionally faster inference. Measurements on real chips contradict that assumption. On NVIDIA Jetson boards, quantizing a model's language backbone to 4 bits cut memory by up to 47.5% but slowed token generation by 10.4% to 56.3%, because unpacking the weights back to higher precision cost more time than the smaller memory transfers saved (Rethinking Small VLM Quantization). We propose to replace the "cost equals bits" assumption in MoE bit allocation with a cost table measured on the target chip. The table is indexed by bit-width and by the one quantity that makes MoE different from dense models: how many tokens an expert actually receives per decoding step. The allocator is then re-solved under a measured latency or energy budget.
The MoE-specific point is this. The time spent unpacking a weight is shared across every token that uses that weight in the same step. A frequently routed "workhorse" expert spreads the unpacking cost over many tokens, while a rarely used expert pays the full cost for one token or none. The best bit-width for an expert therefore depends jointly on the chip and on that expert's traffic. Traffic is exactly the statistic existing allocators use, but they use it only to protect accuracy. Quantization also shifts which experts get chosen, so traffic measured on the full-precision model is the wrong input. If this is right, a single allocator can be dominated on accuracy-versus-measured-speed by a hardware-priced one. If it is wrong, the cheap proxy already ranks configurations correctly, and we will be able to show that directly.
What exists and what is missing
Three bit-allocation papers motivate this work. Each optimizes accuracy against a bit budget, and none times what it produces. MC-MoE solves an integer program that assigns 1, 2 or 3 bits per expert using activation frequency, routing weights and reconstruction error. It reaches 2.54 average bits on Mixtral-8x7B and 52 tokens/s on one RTX 3090, but it never characterizes the cost of running mixed bit-widths across parallel expert kernels. Automated Fine-Grained MoE Quantization shows that down-projection matrices hold 7.58% outlier channels and protects them with channel-level mixed precision. It reports only accuracy, a 3.96% average gain at about 2.2 bits. Examining PTQ for MoE finds that attention, shared experts and early layers need more bits, reaching 65.35% versus 64.30% for uniform GPTQ, and it explicitly defers hardware realization to future work.
Two hardware studies motivate the gap and also constrain the design. MoE-XBench measures a "hardware acceleration gap" on H100 and M1 Max: decode throughput drops by nearly 40% at 32K context when the stored attention cache must be unpacked on the fly. It also warns that average scores hide collapses on coding and tool use, so we evaluate those tasks separately. The Jetson study above supplies the slowdown numbers. Neither study turns its measurements into an allocation method.
Mixture of Cache-Conditional Experts is the closest work in spirit. It biases routing toward experts already held in fast memory and gets a 2x on-device speedup with under 0.1% accuracy loss. It changes routing under uniform quantization. We change bit allocation and take routing as an input that quantization perturbs. The two approaches are complementary, and we test whether they stack.
The methodological template comes from Air Learning. It profiled latency distributions on the target processor and fed them into training, which shrank a 37.73% flight-time gap between simulation and the real drone to 0.5%. DRACO supplies the conceptual warning. When a workload is limited by memory speed, adding arithmetic can reduce latency: it cut latency by 42.6% on a 64×64 systolic array. Bits and arithmetic are therefore poor stand-ins for time.
Core Claim
Let $b_e$ be the bit-width of expert $e$. Let $n_e(b)$ be the expected number of tokens routed to $e$ per decode step under the quantized model $b$. Let $c_h(b_e, n_e)$ be the measured kernel time of that expert on hardware $h$. The allocation problem becomes
$$\min_{b}\ \sum_e \Delta_e(b_e) \quad \text{s.t.} \quad \sum_e c_h\big(b_e, n_e(b)\big) + C_h^{\text{fixed}} \le T,$$
where $\Delta_e$ is the accuracy-loss proxy MC-MoE already uses, $C_h^{\text{fixed}}$ is the measured cost of attention and other non-expert work, and $T$ is the latency budget. Replacing $c_h$ with an energy table gives an energy version. Because $n_e$ depends on $b$, we solve by alternation: allocate, re-measure routing on calibration data, then re-allocate. Our claim is that the measured, traffic-dependent $c_h$ produces configurations that are strictly faster at matched accuracy than bit-priced allocation on at least one platform where unpacking is expensive. We expect the gap to be largest on Jetson and smallest on H100.
Research Program
We first build the cost tables. For each platform (Jetson Orin NX and AGX, RTX 3090, H100), we microbenchmark expert feed-forward kernels at each candidate bit-width and format, sweeping tokens-per-expert from 1 to the largest batch size served. We also include concurrent launches of mixed bit-widths within one layer, which captures the overhead MC-MoE left unmeasured. All methods use one fixed kernel library per platform, so differences come from allocation and not from kernel quality. We report results with a second library to show the conclusions do not hinge on one implementation, given that the Jetson study's 4.66x slowdown came from one library's fragmented kernels.
Next, we measure how far routing drifts under each quantization configuration. We record which top-$k$ selections change and how $n_e$ shifts relative to full precision. Then we solve the allocation problem above and compare it against the alternatives below. On the Snapdragon setting, where experts that do not fit in fast memory must be fetched from flash storage, we add the measured fetch cost to $c_h$ and combine our allocator with cache-prior reranking.
Baselines
The naive reference points are full-precision FP16 and uniform 4-bit weight quantization with the same kernel library. The Jetson study already shows uniform 4-bit can be slower than FP16 there, so "faster than FP16 at negligible accuracy loss" is a meaningful bar.
The strongest existing competitor is MC-MoE's frequency-aware integer-program allocation. It already uses routing statistics and reports a deployed result of 2.54 bits and 52 tokens/s on Mixtral-8x7B on an RTX 3090, with a 1.8x to 2.06x latency speedup over baseline quantization methods. We rerun it on the same hardware, calibration data (512 WikiText sequences, following Examining PTQ for MoE), kernels and solver time budget, and match its average bit-width. Beating it means higher measured decode tokens/s within 0.3 points of its average accuracy on the six-task suite, or higher accuracy at equal measured latency. For accuracy, structure-aware GPTQ allocation (65.35% versus 64.30% uniform on Mixtral) serves as a second allocator baseline.
Two ablations isolate the key ingredients. The first uses a cost table with the tokens-per-expert dimension collapsed to its average, which amounts to a dense-style per-layer latency table. If it matches the full method, the MoE-specific claim fails. The second uses full-precision routing statistics for $n_e$ instead of re-measured quantized routing. This tests whether routing drift matters for cost.
Why not just…?
Why not just use uniform 4-bit with a good kernel and skip mixed precision? This is a real threat. On H100-class hardware with efficient 4-bit kernels, the uniform setting may sit on the measured speed frontier, and mixed precision may add only accuracy. The evidence against uniform 4-bit as a universal answer is the Jetson slowdown and the accuracy gaps reported by the structure-aware allocators. Our experiment settles the question per platform. If uniform 4-bit is never dominated in measured accuracy-versus-latency on any device, we will report that as the finding.
Why not just search configurations directly on the device? Exhaustive on-device search over bit-widths for every expert in every layer is combinatorial, but a coarse search over a few dozen structured configurations (per-component bit-widths) is cheap and may be enough. We include this search, matched for wall-clock profiling time, as a baseline. The full method must beat it at equal profiling time to justify the cost table.
Why not just use an analytic roofline model of memory and compute limits instead of measurements? A roofline model captures the bandwidth-versus-arithmetic tradeoff but omits unpacking pipeline stalls and kernel-launch fragmentation, which are exactly the effects behind the reported slowdowns. We fit a roofline-priced allocator as an ablation. We measure its rank correlation with measured latency, and the regret of the configurations it selects, against the measured table.
Proposed Experiments
We test Mixtral-8x7B and DeepSeek-MoE-16B on RTX 3090 and H100; Qwen-MoE on Snapdragon devices with 12GB and 16GB of memory; and the MoE backbone of Qwen3-VL on Jetson NX and AGX, measuring power from on-board sensors.
For accuracy we use WikiText-2 perplexity and the six-task suite (WinoGrande, COPA, OpenBookQA, HellaSwag, PIQA, MMLU). Following MoE-XBench's warning about averages, we also report HumanEval and BFCLv3 separately, along with MME for the vision-language model.
Systems metrics are decode tokens/s, time per token at batch sizes 1 and 16, and joules per token. The decisive diagnostic comes before any allocation. Across about 200 sampled configurations per platform, we compute the Kendall rank correlation between bit-priced predicted cost and measured latency. If that correlation is near 1 on every platform, the proposal's premise fails and we stop.
If the premise holds, the decisive test is the full method against MC-MoE and the coarse on-device search, at matched accuracy, on Jetson AGX. There the reported slowdowns suggest the largest gap. A positive result there that shrinks on H100 would confirm the mechanism.
References
- Rethinking Small VLM Quantization: From Component-Wise Analysis to Hardware-Aware Edge Deployment
- MC-MoE: Mixture Compressor for Mixture-of-Experts LLMs Gains More
- Automated Fine-Grained Mixture-of-Experts Quantization
- Examining Post-Training Quantization for Mixture-of-Experts: A Benchmark
- Benchmarking Composable Compression Techniques in Mixture-of-Experts LLMs
- Mixture of Cache-Conditional Experts for Efficient Mobile Device Inference
- Air Learning: a deep reinforcement learning gym for autonomous aerial robot visual navigation
- DRACO: Co-Optimizing Hardware Utilization, and Performance of DNNs on Systolic Accelerator