AdaGK-SGD: Adaptive Global Knowledge Guided Distributed Stochastic Gradient Descent

Hangyu Ye, Weiying Xie, Yunsong Li, Leyuan FangAAAI 2025 · DOI

Abstract

Distributed Machine Learning (DML) frequently encounters communication bottlenecks, necessitating local training strategies like Local SGD. While efficient, these methods often suffer from local model drift and overfitting to non-i.i.d. data. This paper introduces AdaGK-SGD, a novel distributed training algorithm that utilizes "global knowledge" to guide local workers throughout the training process. Central to this approach is the Maximum Lifetime of Global Knowledge (MLGK) module, which adaptively determines when global information becomes too stale to be beneficial. Experimental results on CIFAR and ImageNet demonstrate that AdaGK-SGD significantly improves model accuracy and generalization, particularly in communication-constrained environments.

Introduction

In Distributed Machine Learning, communication overhead is the primary bottleneck. To save bandwidth, researchers often employ "local methods" such as Local SGD, where workers train independently for several iterations ($\tau$) before averaging their models. However, this efficiency comes at a cost: local models tend to drift apart or overfit to their specific data subsets.

While "global knowledge" (information shared across the system, like the global average model) can help steer these workers back on track, it suffers from staleness. As local training progresses, the global model from the last synchronization becomes an increasingly poor representation of the current state. AdaGK-SGD addresses this by providing a theoretical framework for determining exactly when this global information stops being helpful and starts being harmful.

Methodology

The AdaGK-SGD framework is structured into three main modules that work in tandem to optimize distributed training:

  1. Module $\mathcal{D}$ (Distributed Strategy): The backbone distributed method, such as Local SGD.
  2. Module $\mathcal{G}$ (Global Knowledge Guidance): This module injects a "correction term" into every local update, using an auxiliary variable to nudge the local model toward the global consensus.
  3. Module $\mathcal{M}$ (Maximum Lifetime - MLGK): The core innovation. Inspired by the "Time-to-Live" (TTL) concept in networking, this module adaptively calculates the Maximum Lifetime ($M$) of global knowledge.
The three-module architecture of AdaGK-SGD, illustrating the interaction between distributed strategy, global guidance, and the MLGK module.

The MLGK module monitors the relationship between global guidance and local gradients. In the convex case, it uses cosine similarity: if the guidance from the global model starts pointing in a direction that conflicts with the local gradient, the global knowledge is deemed "expired." At that point, the algorithm disables the global guidance for the remainder of that local training period to prevent staleness from degrading the model.

Key Results and Insights

The authors evaluated AdaGK-SGD on standard image classification benchmarks, including CIFAR-10, CIFAR-100, and ILSVRC2012 (ImageNet), across various architectures like ResNet and DenseNet.

Performance and Generalization

AdaGK-SGD consistently outperforms standard Local SGD and other state-of-the-art methods. For instance, on CIFAR-100 with ResNet, AdaGK-SGD improved Top-1 accuracy by 1.47% over the Local SGD baseline. On the massive ImageNet dataset, it achieved a 0.502% relative accuracy improvement.

Performance comparison of AdaGK-SGD against baseline methods on CIFAR-10 and CIFAR-100 datasets, demonstrating the effectiveness of the MLGK module across different frameworks.

Plug-and-Play Compatibility

A significant advantage of the MLGK module is its compatibility with existing popular frameworks. The authors demonstrated that integrating MLGK into methods like SlowMo, EASGD, and BMUF-Adam consistently boosts their performance (e.g., MLGK-SlowMo and MLGK-EASGD), as shown in the comparative results.

Scaling with Sparsity

Qualitative analysis shows that the performance gap between AdaGK-SGD and other methods widens as the communication interval ($\tau$) increases. This makes the algorithm particularly valuable for "harsh" network environments where workers can only synchronize rarely. Furthermore, AdaGK-SGD results in a smoother training process compared to the "jittery" convergence curves often seen in other distributed optimizers.

Conclusion and Future Work

AdaGK-SGD provides a robust theoretical and practical foundation for leveraging global knowledge in distributed training. By framing global knowledge as a dynamic regularizer with a finite "lifetime," the algorithm achieves faster convergence and better generalization without increasing communication overhead.

Future research directions include bridging the gap between the theoretical cosine similarity calculations and the simpler heuristic implementations used in practice. Additionally, exploring how AdaGK-SGD interacts with orthogonal techniques like gradient compression or decentralized topologies could further enhance its utility in large-scale distributed systems.

Summary updated

BibTeX
@inproceedings{ye2025adagk,
  author = {Hangyu Ye and Weiying Xie and Yunsong Li and Leyuan Fang},
  title = {{AdaGK-SGD: Adaptive Global Knowledge Guided Distributed Stochastic Gradient Descent}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i21.34365},
}
Figure