No Need to Talk: Asynchronous Mixture of Language Models

Anastasiia Filippova, Angelos Katharopoulos, David Grangier, Ronan Collobert

TL;DR: SMALLTALK LM is a novel framework for training Mixture of Experts (MoE) language models almost entirely asynchronously. By routing entire sequences to specialized, independent experts based on a short initial prefix, it removes the need for expensive high-bandwidth GPU interconnects while achieving significantly better performance than dense models at the same computational cost.

The "Interconnect Tax" in LLM Scaling

Scaling Large Language Models (LLMs) usually comes with a massive "interconnect tax." Standard distributed training relies on synchronous gradient descent, where every GPU must constantly communicate to synchronize weights. This requires specialized, high-bandwidth hardware (like NVIDIA’s NVLink) that is expensive and often inaccessible to smaller research groups.

While Mixture of Experts (MoE) models (like Mixtral or Switch Transformer) offer inference efficiency, they typically route data at the token level. This means every expert must be accessible in memory, and high-speed communication is still required during training. Previous attempts at asynchronous training often relied on pre-clustering the entire dataset or using metadata, which isn't practical for real-world, streaming data applications.

Introducing SMALLTALK LM

The authors introduce SMALLTALK LM, which shifts routing from the token level to the sequence level using a two-stage Expectation-Maximization (EM) approach:

  1. Router Training (Stage 1): Instead of using large experts to decide where data goes, the system trains a set of "tiny" routers (language models with only ~4.4M parameters). These routers learn to specialize in different parts of the data distribution.
  2. Expert Training (Stage 2): Once the routers are trained, the dataset is partitioned. Each expert (a full-sized LM) is trained independently on the data its corresponding router "prefers." Because the experts never need to synchronize gradients, they can be trained on completely separate nodes or even different clusters.
  3. Prefix-Based Routing: At inference time, the system looks at a short prefix of the input (e.g., the first 32 to 256 tokens). The tiny routers score the prefix, and the sequence is sent to the single best-matching expert.

Balanced Assignment Strategy

To ensure no single expert is overwhelmed or left idle, the authors use a balanced assignment strategy during training. Naive assignments can easily lead to imbalanced expert loads. By sorting sequences by their log-likelihood scores within a chunk, the system ensures an even distribution of data across the entire "herd" of experts.

Comparison of naive assignment versus the proposed balanced assignment strategy.

Experimental Setup and Datasets

The models were trained on the RedPajama-V2 dataset, a massive collection derived from 84 Common Crawl snapshots. For evaluation, the authors used standard benchmarks via the lm-eval-harness, including ARC (Challenge and Easy), HellaSwag, SciQ, and MMLU (covering 56 diverse subjects).

Key Insights and Results

  • Router Size is Decoupled from Performance: Surprisingly, the size of the router doesn't significantly impact the final mixture's performance. A tiny 4.4M parameter router performed virtually identically to a full 335M parameter model.
  • Short Prefixes are Sufficient: The system can make highly effective routing decisions using as few as 32 tokens. This makes the approach viable for real-world applications like chatbots or document processing where only the start of a prompt is available initially.
  • Massive Communication Reduction: In a standard setup, a 1.3B parameter model might require transferring 10.4GB of data per node per step. SMALLTALK LM reduces this to just 5.6MB—a thousand-fold reduction in communication overhead.
  • Performance Gains: A 1.3B parameter SMALLTALK LM with 32 experts achieved an 18% improvement in perplexity over a dense baseline trained with the same number of FLOPs. Furthermore, the model outperformed the dense baseline on 75% of downstream tasks (42 out of 56 MMLU tasks).
  • Inference Efficiency: Despite having a much higher total parameter count, the inference cost is nearly identical to a dense model because only one expert is "active" for any given sequence. The routing overhead is negligible (~3%).

Limitations and Future Work

While highly efficient, SMALLTALK LM has limitations. Once the experts are trained on their specific data partitions, the routing is static; the model cannot "switch" experts mid-sequence if the topic of a long document shifts. Additionally, sequence-level routing might miss the finer-grained token-level efficiencies found in traditional MoEs.

Future work will explore scaling this approach to hundreds or even thousands of experts, potentially unlocking even greater specialized performance while maintaining the same sparse inference cost.

Summary updated

Abstract

We introduce SMALLTALK LM, an innovative method for training a mixture of language models in an almost asynchronous manner. Each model of the mixture specializes in distinct parts of the data distribution, without the need of high-bandwidth communication between the nodes training each model. At inference, a lightweight router directs a given sequence to a single expert, according to a short prefix. This inference scheme naturally uses a fraction of the parameters from the overall mixture model. Unlike prior works on asynchronous LLM training, our routing method does not rely on full corpus clustering or access to metadata, making it more suitable for real-world applications. Our experiments on language modeling demonstrate that SMALLTALK LM achieves significantly lower perplexity than dense model baselines for the same total training FLOPs and an almost identical inference cost. Finally, in our downstream evaluations we outperform the dense baseline on 75% of the tasks.
BibTeX
@inproceedings{filippova2025no,
  author = {Anastasiia Filippova and Angelos Katharopoulos and David Grangier and Ronan Collobert},
  title = {{No Need to Talk: Asynchronous Mixture of Language Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=pHOH8FVrTp},
}
Figure