Communicate the Change: Efficient Expert Parallelism via Activation Compression

Fengyuan Liu, Jay Gala, Siva Reddy, Dzmitry BahdanauarXiv 2026 · arXiv

TL;DR

EchoMoE is a lightweight architectural retrofit for Mixture-of-Experts (MoE) models that slashes inter-node communication by 58% and boosts inference throughput by up to 22.6%. By transmitting only the compressed differences between layer activations rather than full hidden states, it overcomes the "communication wall" in distributed LLM serving with minimal impact on model accuracy.

What Problem Does This Paper Address?

As Large Language Models (LLMs) scale, sparse Mixture-of-Experts (MoE) architectures have become the standard for increasing parameter capacity without a massive spike in compute cost. However, these models face a physical limit: the communication bottleneck.

In a distributed setting, MoE layers rely on Expert Parallelism (EP), where different "experts" live on different GPUs or nodes. Every time a token hits an MoE layer, its activation must be "dispatched" to the correct expert and the result "combined" back. As GPU compute power continues to outpace network bandwidth, these synchronous data transfers (often moving hundreds of gigabytes for long sequences) have become the primary drag on inference speed.

Existing solutions often require training a model from scratch or using lower precision. The authors of this paper sought a way to reduce the volume of data moved in existing, pretrained MoE models without sacrificing their "intelligence."

Methods

The core insight of EchoMoE is that Transformer blocks update the residual stream incrementally. This means the hidden state doesn't change much from one layer to the next—nearby layers receive inputs that are highly correlated (often >90% cosine similarity).

Instead of sending the full activation at every layer, EchoMoE does the following:

  1. Reference Caching: It periodically caches a full-width activation from an earlier layer to serve as a "reference."
  2. Delta Communication: For subsequent layers, it calculates the difference (the "delta") between the current state and the reference. It then compresses and transmits only this delta.
  3. Weight Folding: To avoid adding computational overhead, the authors "fold" the linear compression and decompression matrices directly into the existing expert weights. During inference, the experts simply operate on the compressed representations.
  4. Retrofitting: Rather than full pretraining, EchoMoE uses a "retrofitting" process. It initializes the compression parameters to minimize reconstruction error and then performs a brief end-to-end fine-tuning using roughly 8,500× fewer tokens than original pretraining.
Figure 1: EchoMoE reduces EP communication and improves inference throughput. Left: Standard EP dispatches full activations to selected experts and returns full expert outputs to each token’s owner rank. EchoMoE instead compresses dispatch activations using cached cross-layer references and compresses expert outputs before combine. Token A’s activation communication path is highlighted. Right: EchoMoE saves 58% of RDMA bytes and increases throughput by 22.6% at a 3.8 pp task score drop. At similar task performance, EchoMoE sends 26% less RDMA traffic than plain compression, which directly compresses dispatch activations. RDMA savings include reference transfers at EP32 over 4 nodes, with EP communication in BF16. Throughput uses EP32, 128 requests per GPU, 4096 input and 256 output tokens and balanced expert assignment.

Datasets

The authors evaluated EchoMoE across a diverse suite of 18 benchmarks spanning knowledge and reasoning (MMLU, ARC), mathematics (GSM8K, MATH-500), coding (HumanEval+, MBPP+), and instruction following (IFEval).

For the retrofitting process, they used a mixture of the Dolmino and Dolci-Instruct datasets. The method was tested on several prominent MoE models, including Qwen3-30B-A3B, Gemma 4, DeepSeek-V3, and Nemotron-3.5.

Key Insights and Ideas

  • Predictability is the Secret Sauce: The paper quantifies that much of a layer's activation is predictable from an earlier state. Compressing this "residual" is 3.3–3.8× more efficient than compressing the raw activation directly.
  • Expert Outputs are Low-Rank: The "combine" step (expert outputs) is even more compressible than the "dispatch" step (inputs), likely because expert updates to the residual stream have a lower-dimensional structure.
  • Zero-Overhead Inference: Because the decompression is folded into the expert weights, EchoMoE actually reduces the total matrix math required in some configurations, making it a rare "win-win" for both communication and computation.
  • Enabling Wide-Area Inference: By significantly reducing bandwidth requirements, EchoMoE makes it more feasible to run large MoE models over slower networks, like standard Ethernet, where it achieved up to a 2.38× throughput gain.

Key Results

  • Communication Savings: EchoMoE reduced inter-node RDMA traffic by 58% for the Qwen3-30B-A3B model.
  • Throughput Gains: On a cluster of H100 GPUs, inference throughput improved by 22.6%.
  • Accuracy Preservation: These gains came at a small cost to performance, with an average drop of only 3.8 percentage points across all 18 benchmarks.
  • Quantization Friendly: The method composes perfectly with FP8 quantization, seeing no additional quality loss when moving from 16-bit to 8-bit compressed representations.

Limitations and Future Work

While EchoMoE is highly efficient, it does require a small amount of fine-tuning to "retrofit" a pretrained model. The authors also note that as compression becomes more aggressive (e.g., 4× reduction), the performance drop becomes more significant (up to 11.2 pp).

Future work could involve integrating EchoMoE directly into the initial pretraining phase of MoE models, which might allow the model to learn even more compressible representations from the start. Additionally, applying these "delta" communication techniques to the training process itself could significantly speed up the development of the next generation of massive-scale models.

Summary updated

Abstract

Scaling has driven the success of large language models (LLM), with sparse mixture-of-experts (MoE) architectures enabling greater parameter capacity without a proportional increase in computation. At large scales, MoE models rely on expert parallelism to distribute experts across devices. Each MoE layer requires two synchronous collective communication steps during inference: dispatching hidden states to the activated experts and combining the resulting expert outputs. As compute throughput has outpaced interconnect bandwidth, these synchronous collectives have become a major bottleneck to MoE scaling and efficiency. We address this challenge with EchoMoE, a lightweight architecture modification that retrofits existing MoE models, requiring approximately 8,500x fewer training tokens than full pretraining. Motivated by the observation that transformer blocks update the residual stream incrementally, we use a hidden state from an earlier layer as a reference and communicate only the compressed difference between it and the current state, rather than communicating each hidden state in full. EchoMoE periodically refreshes these reference states and transmits only the compressed differences synchronously, reducing the traffic on latency-critical paths without significant computational overhead. EchoMoE reduces inter-node communication volume by 58% and improves throughput by 22.6%, while preserving average performance within 3.8 percentage points across 18 benchmarks. EchoMoE further improves inference throughput by up to 2.38x over bandwidth-constrained Ethernet, paving the way for wide-area distributed inference. EchoMoE also tolerates quantization of the communicated representations without measurable quality loss. We release an efficient implementation for applying EchoMoE to pretrained MoE models, together with an inference integration for vLLM.
BibTeX
@misc{liu2026communicate,
  author = {Fengyuan Liu and Jay Gala and Siva Reddy and Dzmitry Bahdanau},
  title = {{Communicate the Change: Efficient Expert Parallelism via Activation Compression}},
  year = {2026},
  eprint = {2611.22215},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22215},
}
Figure