Distributed Low-Communication Training with Decoupled Momentum Optimization

Sasho Nedelkoski, Alexander Acker, Odej Kao, Sören Becker, Dominik ScheinertarXiv 2025 · arXiv · OpenReview

TL;DR

This paper introduces a novel method for distributed deep learning training that significantly reduces communication overhead by selectively synchronizing only the high-frequency components of the optimizer's momentum. This approach allows for efficient training of large models on networks with limited bandwidth, making distributed training more accessible and flexible.

What Problem Does This Paper Address?

Training massive deep learning models requires immense computational power, typically found in data centers with high-speed, low-latency interconnects. This reliance on specialized infrastructure limits the use of more distributed and potentially cheaper compute resources, especially those with lower bandwidth or geographical separation. Existing methods to reduce communication, like infrequent synchronization or gradient compression, have limitations. This paper aims to bridge this gap by proposing a technique that combines infrequent global synchronization with a smarter way of compressing and synchronizing the information exchanged, specifically focusing on the optimizer's momentum.

Methods

The core innovation lies in "decoupled momentum optimization." The authors build upon the idea of infrequent synchronization (performing multiple local updates before a global sync) and combine it with a signal processing technique: the Discrete Cosine Transform (DCT).

Here's how it works:

  1. Local Training: Each worker node performs several local training steps using a standard optimizer like AdamW.
  2. Momentum Decomposition: After these local steps, instead of synchronizing the entire model or gradients, the optimizer's momentum vector is decomposed into its high- and low-frequency components using the DCT.
  3. Selective Synchronization: Only the top-k high-frequency components of the momentum are synchronized across all worker nodes. The intuition is that these high-frequency components capture the most critical and rapidly changing updates to the model's parameters.
  4. Momentum Reconstruction: Each worker reconstructs its local momentum by combining the synchronized high-frequency components with its own locally retained low-frequency components. The low-frequency components are kept locally to accumulate information over time, preventing permanent loss of gradient information.
  5. Model Update: The model is then updated using this reconstructed momentum.

This approach decouples the high-frequency updates (which are shared frequently) from the low-frequency updates (which are accumulated locally), thereby reducing the amount of data that needs to be communicated globally.

Datasets

The proposed method was evaluated on two standard large-scale datasets:

  • C4 (Colossal Clean Crawled Corpus): Used for training a transformer-based language model (GPT-NeoX).
  • ImageNet-1k: Used for training a convolutional neural network (ResNet).

No new benchmarks were introduced; the focus was on evaluating the existing method on established datasets.

Key Insights and Ideas

  • Momentum as a Signal: The paper treats the optimizer's momentum not just as a state variable but as a signal that can be decomposed and compressed. This perspective opens up new avenues for communication reduction.
  • High-Frequency Dominance: The hypothesis that high-frequency components of momentum are most critical for rapid convergence is key. By synchronizing these, the method aims to quickly align workers on important updates.
  • Decoupling for Flexibility: Separating high-frequency and low-frequency momentum components allows for flexible communication strategies. Low-frequency components can accumulate locally, reducing the need for immediate global synchronization, while high-frequency components ensure rapid progress.
  • DCT as an Efficient Approximation: The Discrete Cosine Transform is used as a practical and computationally efficient alternative to more complex principal component analysis (like KLT) for signal decomposition.
  • Counter-Intuitive Compression Results: For transformer models on the C4 dataset, a higher compression rate (smaller k) sometimes yielded better perplexity, suggesting that the momentum signal in these models might be more robust to compression than initially expected, or that specific properties of the momentum signal are at play.

Key Results

The proposed method, termed "Distributed Low-Communication Training with Decoupled Momentum Optimization," demonstrated significant improvements in communication efficiency:

  • Language Modeling (GPT-NeoX on C4): Achieved up to a 3000x reduction in communication compared to standard Distributed Data Parallel (DDP) with a minimal increase in perplexity (6%). Compared to DiLoCo, it achieved a 16x reduction with a 2.5% perplexity increase.
  • Image Classification (ResNet on ImageNet-1k): Similar communication reductions were observed, with the method outperforming baselines.
  • Generalization: The approach showed consistent benefits across both transformer and CNN architectures.
  • Scalability: The method showed promising results on both 2-node and 4-node setups, with relative trends holding.

Limitations and Future Work

  • Hyperparameter Sensitivity: The optimal value for the k parameter (number of high-frequency components) can vary across models and datasets. The paper notes that while they used optimal parameters for baselines, they did not perform exhaustive hyperparameter optimization for their method due to computational constraints.
  • Theoretical Guarantees: A formal convergence proof for the proposed method is left for future work.
  • Scaling Properties: While evaluated on 2 and 4 nodes, a thorough investigation of scaling properties across a larger number of nodes is needed.
  • Further Communication Mechanisms: Exploring more efficient communication mechanisms beyond simple all-gather operations could further enhance performance.
  • Understanding Compression Effects: Deeper investigation into why certain compression levels work better for specific architectures (like transformers) is planned. This could lead to more principled ways to control compression.

Summary updated

Abstract

The training of large models demands substantial computational resources, typically available only in data centers with high-bandwidth interconnects. However, reducing the reliance on high-bandwidth interconnects between nodes enables the use of distributed compute resources as an alternative to centralized data center training. Building on recent advances in distributed model training, we propose an approach that further reduces communication by combining infrequent synchronizations across distributed model replicas with gradient momentum compression. In particular, we treat the optimizer momentum as a signal and decompose the Nesterov momentum into high- and low-frequency components via the discrete cosine transform (DCT). Only the high-frequency components are synchronized across model replicas every $H$ steps. Empirically, our method achieves up to a $16\times$ reduction in communication compared to the baseline DiLoCo, and it generalizes across architectures, including transformer-based language models and convolutional neural networks for images. Overall, this work advances the feasibility of training large models on distributed nodes with low-bandwidth interconnects.
BibTeX
@misc{nedelkoski2025distributed,
  author = {Sasho Nedelkoski and Alexander Acker and Odej Kao and Sören Becker and Dominik Scheinert},
  title = {{Distributed Low-Communication Training with Decoupled Momentum Optimization}},
  year = {2025},
  eprint = {2510.03371},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.03371},
}
Figure