On Consensus-Optimality Trade-offs in Collaborative Deep Learning

Zhanhong Jiang, Aditya Balu, Chinmay Hegde, Soumik Sarkar

TL;DR

This paper tackles the fundamental challenge in distributed deep learning: balancing the need for agents to agree on a single model (consensus) with the goal of achieving the best possible model performance (optimality). The authors introduce two novel algorithms, i-CDSGD and g-CDSGD, that allow fine-grained control over this trade-off, offering improved performance and consensus compared to existing methods.

What Problem Does This Paper Address?

In distributed deep learning, multiple agents (e.g., devices, servers) collaborate to train a model using their own private data. A key challenge arises from the inherent tension between two desirable properties: consensus and optimality.

  • Consensus: Agents need to agree on a single, unified model. If agents operate too independently, they might converge to different, potentially suboptimal, models.
  • Optimality: The final model should be as good as if it were trained on all the data centrally. However, in a distributed setting, achieving perfect optimality can be hindered by communication constraints and the need for consensus.

Existing methods often struggle to strike the right balance, leading to either models that are not fully converged (poor consensus) or models that are not as accurate as possible (poor optimality). This paper aims to provide algorithms that explicitly manage this trade-off over a fixed communication network topology.

Methods

The paper introduces two novel algorithmic frameworks:

  1. Incremental Consensus-based Distributed Stochastic Gradient Descent (i-CDSGD): This algorithm enhances traditional distributed SGD by incorporating multiple "consensus steps" within each stochastic gradient descent (SGD) iteration. This means agents communicate with their neighbors more frequently within a single update cycle, aiming to push them closer to consensus.
  2. Generalized Consensus-based Distributed SGD (g-CDSGD): This approach offers a more direct way to control the consensus-optimality trade-off. It introduces a tunable parameter ($\omega$) that allows navigation across the entire spectrum, from complete consensus (all agents agree on one model) to complete disagreement (each agent converges to its own individual model, potentially achieving higher local optimality but no consensus).

The authors also develop momentum variants of these algorithms to further accelerate convergence. They provide theoretical convergence guarantees for both strongly convex and non-convex objective functions, as well as for the momentum variants in the strongly convex case.

Datasets

The experiments were conducted using the following datasets:

  • CIFAR-10: A benchmark dataset for image classification.
  • MNIST: A dataset of handwritten digits, also used for image classification.

The paper also explores both balanced and unbalanced data distributions among agents. Unbalanced distributions are particularly challenging as they can exacerbate the consensus problem.

Key Insights and Ideas

  • Explicit Trade-off Control: The core innovation is the development of algorithms (especially g-CDSGD) that allow explicit, tunable control over the balance between consensus and optimality. This moves beyond simply accepting a fixed trade-off.
  • Multi-step Consensus for Heterogeneity: i-CDSGD's incremental consensus steps are shown to be particularly beneficial in heterogeneous settings where agents might have different data distributions or learning capabilities, helping them converge to a common solution more effectively than single-step consensus.
  • Momentum for Speed: The momentum variants of the proposed algorithms demonstrate faster convergence rates, a common technique in optimization that is effectively adapted here for the distributed setting.
  • Consensus Improves Robustness: Even when aiming for high accuracy, achieving better consensus (as demonstrated by lower accuracy differences between agents) leads to more stable training and potentially better generalization.

Key Results

  • Improved Consensus: Both i-CDMSGD and g-CDMSGD (with appropriate parameter tuning) achieve significantly better consensus (lower accuracy difference between best and worst agents) compared to standard distributed SGD (CDMSGD), especially in unbalanced data scenarios.
  • Competitive Accuracy: Despite improving consensus, the proposed algorithms maintain competitive average accuracy, often matching or exceeding state-of-the-art methods like CDMSGD.
  • Faster Convergence with Momentum: The momentum variants (i-CDMSGD and g-CDMSGD) show faster convergence to a stable accuracy level.
  • g-CDMSGD Flexibility: The parameter $\omega$ in g-CDMSGD allows for fine-tuning. Higher $\omega$ values (closer to 1) lean towards individual optimality but can slow convergence, while lower $\omega$ values (closer to 0) prioritize consensus, potentially at the cost of a slightly larger generalization gap.
  • Outperforming Federated Averaging: The proposed methods significantly outperform Federated Averaging in terms of average accuracy.

Limitations and Future Work

  • Fixed Topology: The current algorithms assume a static communication topology. Future work could explore learning with dynamic or time-varying networks.
  • Non-uniform Data Distributions: While the paper addresses unbalanced data, more complex non-uniform distributions could be a focus for future research.
  • Hyperparameter Sensitivity: The performance of g-CDMSGD is influenced by the $\omega$ parameter, and finding the optimal value might require tuning.
  • Local Minima: For non-convex problems, the algorithms are proven to converge to first-order stationary points, but avoiding local minima and saddle points remains a general challenge in deep learning.

The authors suggest that future research directions include learning with non-uniform data distributions and time-varying networks.

Summary updated

Abstract

p>In distributed machine learning, where agents collaboratively learn from diverse private data sets, there is a fundamental tension between <italic>consensus</italic> and <italic>optimality</italic>. In this paper, we build on recent algorithmic progresses in distributed deep learning to explore various consensus-optimality trade-offs over a fixed communication topology. First, we propose the <italic>incremental consensus</italic>-based distributed stochastic gradient descent (i-CDSGD) algorithm, which involves multiple consensus steps (where each agent communicates information with its neighbors) within each SGD iteration. Second, we propose the <italic>generalized consensus</italic>-based distributed SGD (g-CDSGD) algorithm that enables us to navigate the full spectrum from complete consensus (all agents agree) to complete disagreement (each agent converges to individual model parameters). We analytically establish convergence of the proposed algorithms for strongly convex and nonconvex objective functions; we also analyze the momentum variants of the algorithms for the strongly convex case. We support our algorithms via numerical experiments, and demonstrate significant improvements over existing methods for collaborative deep learning.</p>
BibTeX
@inproceedings{jiang2021consensus,
  author = {Zhanhong Jiang and Aditya Balu and Chinmay Hegde and Soumik Sarkar},
  title = {{On Consensus-Optimality Trade-offs in Collaborative Deep Learning}},
  booktitle = {Frontiers Artif. Intell},
  year = {2021},
  url = {https://openreview.net/forum?id=elH81zDhdh},
}
Figure