On Consensus-Optimality Trade-offs in Collaborative Deep Learning

Zhanhong Jiang, Aditya Balu, Chinmay Hegde, Soumik Sarkar

Abstract

In decentralized deep learning, agents must balance the need for consensus (agreement on a model) with the pursuit of optimality (the best model for the collective data). This paper introduces two algorithmic frameworks, i-CDSGD and g-CDSGD, designed to navigate this trade-off through tunable parameters. These methods, along with their momentum-based variants, provide enhanced stability and better performance, particularly in non-IID data environments.

Introduction

Collaborative deep learning enables multiple agents—such as mobile devices or autonomous robots—to train models on private local data while communicating over a network. A core difficulty in this setting is the "consensus-optimality trade-off." Without sufficient communication, agents' models diverge toward their respective local optima, losing the benefits of collective learning. Conversely, excessive communication can lead to prohibitive network overhead.

Existing decentralized optimization methods often struggle to maintain this balance. This research seeks to provide a theoretical and practical framework for managing these competing objectives, ensuring that agents can reach a high degree of agreement without sacrificing global model accuracy.

The optimization landscape: agents move toward local optima ($\theta^*_i$) while aiming for a consensus point ($\hat{\theta}$) near the global optimum ($\theta_*$).

Algorithmic Frameworks

The authors propose two extensions to the standard Consensus Distributed SGD (CDSGD):

1. Incremental Consensus (i-CDSGD)

In i-CDSGD, agents perform $\tau$ consensus steps for every single gradient update. This "incremental" approach forces agents into closer agreement before they proceed with further local training, effectively tightening the consensus bound at the cost of increased communication frequency.

2. Generalized Consensus (g-CDSGD)

g-CDSGD introduces a tuning parameter, $\omega \in (0,1]$, which allows for a weighted update that balances the consensus value with the local gradient step. This "knob" allows practitioners to explicitly prioritize either agreement (low $\omega$) or local learning speed (high $\omega$).

Furthermore, the authors develop momentum variants (i-CDMSGD and g-CDMSGD) and provide the first rigorous convergence proofs for momentum-based consensus algorithms in both strongly convex and non-convex settings.

Theoretical Comparison

The proposed methods are evaluated against state-of-the-art decentralized optimization techniques. As summarized in the table below, i-CDSGD and g-CDSGD offer competitive theoretical bounds for consensus and optimality error while supporting features like momentum analysis and stochastic gradients.

Theoretical comparison of distributed optimization methods, highlighting the convergence rates and features of the proposed frameworks.

Empirical Results and Insights

The frameworks were tested on CIFAR-10 and MNIST benchmarks under both balanced and unbalanced (Non-IID) data distributions. Key findings include:

  • Improved Stability: Momentum-based variants (i-CDMSGD and g-CDMSGD) exhibited significantly smoother performance curves and lower accuracy fluctuations across training epochs compared to standard CDSGD.
  • Robustness to Unbalanced Data: In scenarios where data is distributed unevenly, standard methods often fail to achieve strong consensus. The proposed methods effectively reduced the performance gap between the best and worst-performing agents.
  • Generalization Benefits: g-CDMSGD was observed to have a smaller generalization gap, suggesting that the consensus-driven update mechanism may serve as a form of regularization that improves test-time performance.

Conclusion and Future Work

The i-CDSGD and g-CDSGD frameworks provide a robust way to manage the consensus-optimality trade-off in decentralized learning. While i-CDSGD increases communication overhead, g-CDSGD offers a flexible alternative through its weighting parameter. Future research will focus on scaling these methods to larger network topologies and adapting them for dynamic, time-varying communication graphs.

Summary updated

Abstract

In distributed machine learning, where agents collaboratively learn from diverse private data sets, there is a fundamental tension between consensus and optimality. In this paper, we build on recent algorithmic progresses in distributed deep learning to explore various consensus-optimality trade-offs over a fixed communication topology. First, we propose the incremental consensus-based distributed SGD (i-CDSGD) algorithm, which involves multiple consensus steps (where each agent communicates information with its neighbors) within each SGD iteration. Second, we propose the generalized consensus-based distributed SGD (g-CDSGD) algorithm that enables us to navigate the full spectrum from complete consensus (all agents agree) to complete disagreement (each agent converges to individual model parameters). We analytically establish convergence of the proposed algorithms for strongly convex and nonconvex objective functions; we also analyze the momentum variants of the algorithms for the strongly convex case. We support our algorithms via numerical experiments, and demonstrate significant improvements over existing methods for collaborative deep learning.
BibTeX
@inproceedings{jiang2021consensus,
  author = {Zhanhong Jiang and Aditya Balu and Chinmay Hegde and Soumik Sarkar},
  title = {{On Consensus-Optimality Trade-offs in Collaborative Deep Learning}},
  booktitle = {Frontiers Artif. Intell},
  year = {2021},
  url = {https://openreview.net/forum?id=elH81zDhdh},
}
Figure