Wide, Sparse, Low-Bit Token Codes for Expert-Parallel Communication

Large mixture-of-experts (MoE) language models are made of many small "expert" sub-networks spread across GPUs. A learned gate picks a few experts for each token. Every token then has to be shipped to the GPUs holding its chosen experts and shipped back afterward. This exchange is an all-to-all transfer, where every GPU sends data to every other GPU. It happens twice per MoE layer in the forward pass and twice more in the backward pass. It cannot start until the gate has decided, so it is hard to hide behind computation, and it gets worse across slower inter-node links. Current fixes either shrink what is sent with a dense linear projection before sending, or overlap the transfer with other work. The tokens always travel as dense 16-bit vectors.

We propose sending each token as a wide but mostly-zero, low-precision code instead. Before dispatch, a learned projection maps the token into a latent that is several times wider than a compressed dense latent would be, and a squared-ReLU nonlinearity makes most of that latent zero. We then force the survivors into a fixed hardware-friendly sparsity pattern ("N:M sparsity": at most N nonzeros in every group of M consecutive values) and store them in 8-bit or 4-bit format. Because the pattern is fixed, the index metadata has a known, small size. Any pattern with at most 2 nonzeros per 16 also satisfies the 2-of-4 pattern that NVIDIA sparse Tensor Cores accelerate. The receiving expert can therefore feed the packet straight into a sparse matrix multiply without first expanding it to dense 16-bit. The claim is simple and falsifiable: at equal bytes on the wire, a wide sparse low-bit code preserves more of what the experts need than a narrow dense low-bit code. If that holds, it lowers communication and expert compute together.

Core Claim

The closest prior work compresses MoE communication in one dimension at a time. BigMac moves the down-projection before dispatch ("descend, then communicate"). It reports roughly 75% less network traffic at a downscaling factor of 0.25 and up to 3.07× faster training. The communicated latent is still dense and 16-bit. Multi-Head LatentMoE and Head Parallel splits tokens into sub-tokens and routes them deterministically. It cuts inter-GPU volume to 25% of standard expert parallelism, but it runs in BF16, sparsity plays no part, and it was measured only on single nodes. CoCoQuant keeps activations in low precision through AllReduce and AllGather to avoid repeated quantize/dequantize casts. It does not touch the dynamic, point-to-point dispatch and combine of expert parallelism, and it uses no sparsity.

On the sparsity side, Accelerating Transformer Inference and Training with 2:4 Activation Sparsity shows three things: squared-ReLU activations become 50–98% sparse during training, those zeros can be forced into a 2:4 pattern for Tensor Core speedups, and perplexity is unchanged. This work lives inside a single dense feed-forward layer and never crosses a network.

Our proposal changes what the communicated tensor is. It becomes a nonlinear sparse code whose support pattern itself carries information. That makes it closer to a learned codec than to a linear projection. The prior results each contribute a piece of the case. Rate-Distortion Optimization for Transformer Inference found that nonlinear learned codecs beat linear basis projections by 10.7% in BD-rate on split-inference activations, which motivates a nonlinear code. The 2:4 activation-sparsity paper supplies the mechanism for making such a code hardware-usable. CoCoQuant supplies the principle of keeping one precision along the critical path.

A back-of-envelope rate calculation shows why the regime is interesting. A dense latent of width $d$ at $b$ bits costs $db/8$ bytes per token. A 2:16 sparse latent of width $w$ carries $w/8$ nonzeros, each with $b$ value bits and a 4-bit position inside its group of 16, so it costs $(w/8)(b+4)/8$ bytes. At $b=8$, equal bytes means $w \approx 5.3d$. In other words, the sparse code can be about five times wider with one-eighth of its entries active. Whether that trade favors the sparse code is the central empirical question.

Research Program

We will build an MoE layer with three stages. In encode, on the token's home GPU, the token is projected up to a wide latent of width $w$ and passed through squared ReLU. We keep the top N entries in each group of M, and quantize the survivors to FP8 or INT4 with per-group scales. In exchange, a custom all-to-all packs values, fixed-width indices, and scales into contiguous buffers. In expert, the receiver converts the indices into 2:4 metadata and runs the expert's first matrix multiply as a sparse low-bit GEMM, with no dense 16-bit intermediate. The token permutation trick from the 2:4 activation paper is reused so that the sparse operand lands in the position the hardware expects.

The combine direction returns a down-projected expert output. Squared ReLU is wrong there because signs matter. Instead we keep the top-magnitude entries per group with a straight-through gradient. To stabilize these masks against flicker, we borrow the refined estimator from Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch.

This design gives a benefit in training that dense compression lacks. The forward mask fixes which latent coordinates carry gradient, so the backward all-to-all can reuse the forward index metadata and send only values. The backward pass never ships dense gradients across the network. Prior sparse-training work such as BAME removes the dense-backward cost only on a single device, so this is the part of the gap our design addresses.

The second component is a per-layer rate allocator. It chooses the latent width, the sparsity pattern (1:16, 2:16, 2:8), and the value precision for each MoE layer under a total byte budget. Two findings motivate letting rate vary with depth. The rate-distortion study found that deeper Transformer activations need much higher bitrates: on GPT-2, moving the split point from layer 3 to layer 9 raised the required rate from about 148 to about 492 bits per token. DSMoE reports a W-shaped profile of expert usage across layers. We will start with a cheap first-order sensitivity estimate in the style of Distribution-Aware Adaptive Multi-Bit Quantization. The resulting budget problem will be solved with a small integer program, as CoCoQuant does for its precision graph. The allocator is secondary. The core claim must hold at uniform per-layer settings before the allocator is credited with anything.

Baselines

The naive reference is standard expert parallelism with dense BF16 dispatch and combine. It is trained with the same data, token count, expert count, top-k, and cluster. This tells us how much communication time there is to remove.

The first simple competitor quantizes the unmodified dispatch and combine tensors to FP8 with per-token scales and changes nothing else. It is the cheapest thing a practitioner would try, and it caps the gain from precision alone at about 2× over BF16.

The strongest existing structural method is BigMac's descend-before-communicate design. We will reproduce it at its reported factor of 0.25. BigMac reports reaching validation perplexity 13.69 in 22.8 hours versus 38.9 hours for its baseline, with about 75% lower traffic. We will also run Multi-Head LatentMoE with Head Parallel, which reports a zero-shot average of 52.49% versus 50.00% for standard expert parallelism on 50B FineWeb-Edu tokens, along with a 1.82× training speedup at 4B parameters.

The decisive ablation is BigMac-plus-FP8 at matched bytes. This is a dense latent narrowed until its wire size equals our sparse code's, trained with the same tuning budget (the same learning-rate sweep of five values for every arm). Two further ablations isolate each ingredient. The first uses unstructured top-k selection with explicit coordinate lists at equal bytes, which tests whether the fixed N:M structure costs quality. The second keeps our sparse code in BF16, which tests whether low-bit values matter once sparsity is present.

To beat the field, our method must satisfy two conditions at equal wall-clock or equal tokens. It must match or beat the matched-byte dense-FP8 arm on validation loss and zero-shot average. It must also cut measured all-to-all time by more than that arm does, because the receiver-side sparse GEMM saves compute in addition to bytes.

Why not just…?

Why not just make BigMac's dense projection narrower and send it in FP8? This is the real threat. It is exactly the matched-byte ablation above. If a dense narrow FP8 latent matches our quality at equal bytes, then the claimed advantage of a sparse nonlinear code over a linear one does not survive. The file offers indirect support for our side: the rate-distortion paper's 10.7% BD-rate gain of learned nonlinear codecs over linear bases. That result is on split inference, not MoE, so we treat it as motivation, not evidence. If the dense arm wins, the honest conclusion is that sparsity should be used only for its compute benefit at the receiver, and we would report that.

Why not just overlap communication with computation, as in Shortcut-connected expert parallelism or FoldMoE? Shortcut-connected Expert Parallelism routes early using the previous layer's activations, and FoldMoE interleaves attention and MoE blocks. FoldMoE itself documents that expert computation is too small to mask transfers for long sequences or over inter-node links. Lita stresses that all-to-all is blocking, so any excess communication stalls the GPU. Overlap and compression are also complementary. We will run shortcut-style early routing with and without our codes on a two-node configuration. The test is whether the residual exposed communication time goes to near zero only when the two are combined.

Why not just threshold squared-ReLU activations and send them in a standard sparse format? Unstructured sparsity needs per-entry coordinates. Those coordinates consume much of the saving, and the variable length makes buffer packing irregular. General sparse libraries also need very high sparsity to beat dense kernels on GPUs, as Accelerating Sparse Neural Networks on Dense Hardware reviews. The unstructured top-k ablation measures both costs directly: wire bytes including indices, and receiver GEMM time.

Proposed Experiments

We will pre-train MoE models at roughly 1B and 4B total parameters. The data is the 50B-token FineWeb-Edu subset used by Multi-Head LatentMoE, drawn from The FineWeb Datasets, so our zero-shot suite matches theirs: HellaSwag, PIQA, LAMBADA, ARC-Easy, and ARC-Challenge. The squared-ReLU design choice will be sanity-checked against the reference result that a 1.5B squared-ReLU model with 2:4 activation sparsity reached perplexity 2.652 versus 2.654 for dense SwiGLU on DCLM.

Runs will use a single node and two to four nodes over InfiniBand. The multi-node regime is untested for Head Parallel and is where communication hurts most. The metrics are bytes per token per layer, measured all-to-all time forward and backward, step time, tokens per second, validation loss at matched tokens and at matched wall-clock, and zero-shot average.

Three checks decide the proposal. The first is the matched-byte comparison against dense FP8 latents, swept over at least three byte budgets to trace a quality-versus-bytes curve. The sparse code must dominate this curve, not win at a single point. The second measures receiver-side GEMM time with the packed sparse operand against a dense FP8 GEMM on the unpacked latent. It must hold on hardware that actually supports sparse matmul at the chosen precision, and we will verify this before claiming speed. The third measures backward all-to-all bytes with metadata reuse against dense-gradient transfer. The per-layer allocator is evaluated last, against the best uniform setting at the same total budget. It earns its place only if depth-varying rates lower loss at equal bytes.