Abstract
Distributed Machine Learning (DML) frequently encounters communication bottlenecks, necessitating local training strategies like Local SGD. While efficient, these methods often suffer from local model drift and overfitting to non-i.i.d. data. This paper introduces AdaGK-SGD, a novel distributed training algorithm that utilizes "global knowledge" to guide local workers throughout the training process. Central to this approach is the Maximum Lifetime of Global Knowledge (MLGK) module, which adaptively determines when global information becomes too stale to be beneficial. Experimental results on CIFAR and ImageNet demonstrate that AdaGK-SGD significantly improves model accuracy and generalization, particularly in communication-constrained environments.
Introduction
In Distributed Machine Learning, communication overhead is the primary bottleneck. To save bandwidth, researchers often employ "local methods" such as Local SGD, where workers train independently for several iterations ($\tau$) before averaging their models. However, this efficiency comes at a cost: local models tend to drift apart or overfit to their specific data subsets.
While "global knowledge" (information shared across the system, like the global average model) can help steer these workers back on track, it suffers from staleness. As local training progresses, the global model from the last synchronization becomes an increasingly poor representation of the current state. AdaGK-SGD addresses this by providing a theoretical framework for determining exactly when this global information stops being helpful and starts being harmful.
Methodology
The AdaGK-SGD framework is structured into three main modules that work in tandem to optimize distributed training:
- Module $\mathcal{D}$ (Distributed Strategy): The backbone distributed method, such as Local SGD.
- Module $\mathcal{G}$ (Global Knowledge Guidance): This module injects a "correction term" into every local update, using an auxiliary variable to nudge the local model toward the global consensus.
- Module $\mathcal{M}$ (Maximum Lifetime - MLGK): The core innovation. Inspired by the "Time-to-Live" (TTL) concept in networking, this module adaptively calculates the Maximum Lifetime ($M$) of global knowledge.
The MLGK module monitors the relationship between global guidance and local gradients. In the convex case, it uses cosine similarity: if the guidance from the global model starts pointing in a direction that conflicts with the local gradient, the global knowledge is deemed "expired." At that point, the algorithm disables the global guidance for the remainder of that local training period to prevent staleness from degrading the model.
Key Results and Insights
The authors evaluated AdaGK-SGD on standard image classification benchmarks, including CIFAR-10, CIFAR-100, and ILSVRC2012 (ImageNet), across various architectures like ResNet and DenseNet.
Performance and Generalization
AdaGK-SGD consistently outperforms standard Local SGD and other state-of-the-art methods. For instance, on CIFAR-100 with ResNet, AdaGK-SGD improved Top-1 accuracy by 1.47% over the Local SGD baseline. On the massive ImageNet dataset, it achieved a 0.502% relative accuracy improvement.
Plug-and-Play Compatibility
A significant advantage of the MLGK module is its compatibility with existing popular frameworks. The authors demonstrated that integrating MLGK into methods like SlowMo, EASGD, and BMUF-Adam consistently boosts their performance (e.g., MLGK-SlowMo and MLGK-EASGD), as shown in the comparative results.
Scaling with Sparsity
Qualitative analysis shows that the performance gap between AdaGK-SGD and other methods widens as the communication interval ($\tau$) increases. This makes the algorithm particularly valuable for "harsh" network environments where workers can only synchronize rarely. Furthermore, AdaGK-SGD results in a smoother training process compared to the "jittery" convergence curves often seen in other distributed optimizers.
Conclusion and Future Work
AdaGK-SGD provides a robust theoretical and practical foundation for leveraging global knowledge in distributed training. By framing global knowledge as a dynamic regularizer with a finite "lifetime," the algorithm achieves faster convergence and better generalization without increasing communication overhead.
Future research directions include bridging the gap between the theoretical cosine similarity calculations and the simpler heuristic implementations used in practice. Additionally, exploring how AdaGK-SGD interacts with orthogonal techniques like gradient compression or decentralized topologies could further enhance its utility in large-scale distributed systems.
Summary updated