Quantizing Mixture of Experts Transformers

16 concepts16 papers1 proposal

Quantization is the cornerstone of modern large language model (LLM) deployment. By compressing 16-bit floating-point weights and activations down to 4-bit or even lower formats, practitioners can squeeze massive models onto consumer hardware and achieve dramatic throughput speedups.

In standard dense Transformer architectures, quantization strategies rely on predictable statistical regularities: activations follow continuous distributions, systematic outliers can be mathematically isolated or smoothed, and layer-wise sensitivity scales predictably with training duration.

However, a major architectural shift is underway. To scale parameter counts efficiently, modern architectures increasingly rely on Mixture-of-Experts (MoE) structures. Instead of routing every token through a uniform sequence of dense layers, MoEs use a dynamic routing mechanism to dispatch tokens to a sparse subset of specialized "expert" networks. This architectural divergence introduces a fundamental friction: the activation space becomes highly discontinuous, parameter utilization patterns change dynamically, and small quantization errors in the routing logic can completely alter a token's computational path.

Understanding why compression techniques fail to generalize directly from dense models to MoEs requires looking closely at how sparse routing breaks the core assumptions of modern quantization frameworks.

Why Outlier Mitigation Fails in MoEs

In dense models, the primary culprit behind quantization error is the presence of systematic activation outliers—features with disproportionately large magnitudes that emerge in specific channels. If you quantize these activations uniformly, the massive range dictated by the outliers destroys the numerical resolution for normal values, leading to severe performance degradation.

Advanced post-training quantization (PTQ) frameworks solve this by transforming the activation space to make it more "quantization-friendly." For instance, FPTQuant introduces lightweight, function-preserving mathematical transforms that smooth out extreme values directly within the weights. Similarly, QuaRot applies computational invariance via orthogonal rotations to smear outliers uniformly across channels, while QUIK isolates a tiny fraction of outlier features in higher precision while keeping the rest in 4-bit. Other approaches, such as OSTQuant and TwinQuant, use learnable scaling or subspace decomposition to maximize quantization space utilization.

Dense Transformer:
[Token] ---> [Uniform Layer] (Outliers are systematic and continuous across tokens)
              |__ Managed by Global Rotations / Smoothing Transforms

Sparse MoE Transformer:
[Token] ---> [Router] ---> Expert 1 (Conditional Activation Distribution A)
                      ---> Expert 2 (Conditional Activation Distribution B)

When applied to MoEs, these techniques encounter a structural wall. Because tokens are dynamically routed, the activation distributions entering each expert are highly conditional and sparse. As noted by van Breugel et al. in FPTQuant and Alistarh et al. in QuaRot, mathematical transforms cannot simply be baked into weights globally; they must interact with the modular, conditional routing mechanisms. If an outlier mitigation strategy changes the coordinate system of the activations before the router, it risks corrupting the routing decisions themselves. Each expert effectively sees a distinct, truncated distribution of tokens, meaning that a global smoothing or rotation matrix optimized for the average token distribution will fail to fit the specialized activation characteristics of individual experts.

The Sensitivity of Dynamic Routing

Quantization error is rarely uniform across a model's depth. In dense networks, theoretical frameworks like the linearity theorem utilized by Panferov et al. in HIGGS allow practitioners to predict how precision loss propagates through layers based on structural sensitivity. Furthermore, Catalan-Tatjer et al. in Training Dynamics Impact Post-Training Quantization Robustness and Ouyang et al. in Low-Bit Quantization Favors Undertrained LLMs demonstrate that a model's sensitivity to low-bit quantization is deeply tied to its training dynamics. As dense models are exposed to more tokens, their representations become highly optimized and rigid, making well-trained models paradoxically more brittle to precision drops than undertrained ones.

In sparse architectures, this sensitivity is compounded by the discrete nature of routing. In a dense layer, a minor perturbation in a weight matrix causes a minor perturbation in the output magnitude. In an MoE layer, that same minor perturbation can flip a routing decision entirely.

Because the router typically selects the top-$k$ experts based on a softmax distribution over expert scores, a tiny quantization error can shift the ranking boundary. This causes a token to bypass its optimal expert and route to an unspecialized one, leading to catastrophic functional collapse. Frameworks like the Discarded Weight Recycling (DWR) method proposed by Yu et al. in Treasures in Discarded Weights for LLM Quantization attempt to reclaim quantization noise and merge it back into the model. However, how these "recycled" errors interact with the non-linear, discrete boundaries of MoE routing functions remains an open question.

The Calibration and Pruning Bottleneck

Data-aware compression methods rely on a small set of calibration data to map out a model's activation space and determine which weights are safe to compress or discard. For example, the COLA framework introduced by He et al. in Preserving LLM Capabilities through Calibration Data Curation: From Analysis to Optimization curates calibration data based on representativeness and diversity in the activation space to protect complex reasoning capabilities. Similarly, structured pruning frameworks like NIRVANA, developed by Ai et al. in NIRVANA: Structured pruning reimagined for large language models compression, use neural tangent kernels to identify and remove redundant parameters while preserving hardware efficiency.

Calibration Data Space:
[Standard Dataset] ---> Activates Experts 1, 2, 3 heavily.
                   ---> Experts 4, 5, 6 remain dormant.
                   (Result: Inaccurate sensitivity profiles for dormant experts)

For MoE models, standard calibration data curation and pruning metrics break down. Because different experts specialize in different semantic or syntactic domains, a standard calibration dataset may completely fail to activate certain experts. If an expert remains dormant during calibration, the quantization or pruning framework will miscalculate its sensitivity profile, leading to aggressive compression that destroys its specialized capabilities. As Ai et al. point out, structured pruning cannot be directly translated to MoEs without explicitly accounting for the unique parameter distributions and routing frequencies across experts.

Scaling Alternative Compression Paradigms

To push past the limits of standard uniform quantization, researchers have explored exotic compression spaces:

The open frontier for all these paradigms is scalability to sparse architectures. Whether decomposing subspaces, shifting to the frequency domain, or constraining fine-tuning transformations, every method must eventually confront the fact that an MoE is not a single monolithic matrix, but a dynamic, heterogeneous ecosystem of experts. Bridging this gap requires compression algorithms that treat the router and its experts not as standard linear layers, but as a co-dependent, conditional system.

Key Foundational Work

Treasures in Discarded Weights for LLM Quantization (2025) — contributed 1 concept elements to this direction.
QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs (2024) — contributed 1 concept elements to this direction.
OSTQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fitting (2025) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_sparse_architecture_quantization_3a43beda,
  title = {sparse architecture quantization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sparse-architecture-quantization/rdr_3a43bedad9d9463b90613f7b9b480bbb},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_sparse_architecture_quantization_3a43beda_v1,
  title = {sparse architecture quantization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sparse-architecture-quantization/rdr_3a43bedad9d9463b90613f7b9b480bbb/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake