Abstract
In decentralized deep learning, agents must balance the need for consensus (agreement on a model) with the pursuit of optimality (the best model for the collective data). This paper introduces two algorithmic frameworks, i-CDSGD and g-CDSGD, designed to navigate this trade-off through tunable parameters. These methods, along with their momentum-based variants, provide enhanced stability and better performance, particularly in non-IID data environments.
Introduction
Collaborative deep learning enables multiple agents—such as mobile devices or autonomous robots—to train models on private local data while communicating over a network. A core difficulty in this setting is the "consensus-optimality trade-off." Without sufficient communication, agents' models diverge toward their respective local optima, losing the benefits of collective learning. Conversely, excessive communication can lead to prohibitive network overhead.
Existing decentralized optimization methods often struggle to maintain this balance. This research seeks to provide a theoretical and practical framework for managing these competing objectives, ensuring that agents can reach a high degree of agreement without sacrificing global model accuracy.
Algorithmic Frameworks
The authors propose two extensions to the standard Consensus Distributed SGD (CDSGD):
1. Incremental Consensus (i-CDSGD)
In i-CDSGD, agents perform $\tau$ consensus steps for every single gradient update. This "incremental" approach forces agents into closer agreement before they proceed with further local training, effectively tightening the consensus bound at the cost of increased communication frequency.
2. Generalized Consensus (g-CDSGD)
g-CDSGD introduces a tuning parameter, $\omega \in (0,1]$, which allows for a weighted update that balances the consensus value with the local gradient step. This "knob" allows practitioners to explicitly prioritize either agreement (low $\omega$) or local learning speed (high $\omega$).
Furthermore, the authors develop momentum variants (i-CDMSGD and g-CDMSGD) and provide the first rigorous convergence proofs for momentum-based consensus algorithms in both strongly convex and non-convex settings.
Theoretical Comparison
The proposed methods are evaluated against state-of-the-art decentralized optimization techniques. As summarized in the table below, i-CDSGD and g-CDSGD offer competitive theoretical bounds for consensus and optimality error while supporting features like momentum analysis and stochastic gradients.
Empirical Results and Insights
The frameworks were tested on CIFAR-10 and MNIST benchmarks under both balanced and unbalanced (Non-IID) data distributions. Key findings include:
- Improved Stability: Momentum-based variants (i-CDMSGD and g-CDMSGD) exhibited significantly smoother performance curves and lower accuracy fluctuations across training epochs compared to standard CDSGD.
- Robustness to Unbalanced Data: In scenarios where data is distributed unevenly, standard methods often fail to achieve strong consensus. The proposed methods effectively reduced the performance gap between the best and worst-performing agents.
- Generalization Benefits: g-CDMSGD was observed to have a smaller generalization gap, suggesting that the consensus-driven update mechanism may serve as a form of regularization that improves test-time performance.
Conclusion and Future Work
The i-CDSGD and g-CDSGD frameworks provide a robust way to manage the consensus-optimality trade-off in decentralized learning. While i-CDSGD increases communication overhead, g-CDSGD offers a flexible alternative through its weighting parameter. Future research will focus on scaling these methods to larger network topologies and adapting them for dynamic, time-varying communication graphs.
Summary updated