Decentralized SGD with Controlled Disagreement Finds Flatter Minima

Zesen Wang, Mikael Johansson

TL;DR

Traditionally, "consensus errors"—the disagreements between individual workers—in decentralized deep learning are viewed as harmful noise to be minimized. This paper flips the script, demonstrating that controlled worker disagreement acts as a free implicit regularizer that guides models toward flatter minima. By introducing Decentralized SGD with Adaptive Consensus (DSGD-AC), the authors strategically maintain these errors throughout training, outperforming both standard decentralized SGD and centralized SGD with zero extra computational overhead.

What Problem Does This Paper Address?

In decentralized training, multiple workers collaborate to train a model by exchanging parameters only with their immediate neighbors, avoiding the massive communication bottleneck of a central server. However, because communication is localized, workers' models inevitably drift apart, creating "consensus errors."

Conventional wisdom dictates that these discrepancies undermine convergence and generalization, leading researchers to design complex topologies and algorithms specifically to eliminate them. While recent theories suggested that consensus errors might act as random perturbations that mimic Sharpness-Aware Minimization (SAM)—a technique that explicitly seeks out robust, "flat" minima in the loss landscape—standard Decentralized SGD (DSGD) suffers from a major flaw: as the learning rate decays over time, the consensus errors naturally vanish. This means the beneficial regularization effect disappears precisely during the final, critical stages of training.

Methods

The authors introduce Decentralized SGD with Adaptive Consensus (DSGD-AC) to prevent consensus errors from collapsing.

Instead of keeping the consensus step-size weight constant, DSGD-AC introduces a time-varying scaling factor: $$\gamma^{(t)} = \left[\frac{\alpha^{(t)}}{\alpha_{\max}}\right]^p$$ where $\alpha^{(t)}$ is the decaying learning rate and $p$ is a tuning hyperparameter (typically set to 3). By scaling down the consensus strength ($\gamma$) alongside the learning rate, the algorithm prevents the consensus regularizer from completely dominating the objective function late in training.

Norm of consensus errors evaluated across epochs

Through a mathematical "modal analysis," the authors prove that decaying $\gamma$ selectively weakens graph-based model blending (which forces workers together) while leaving curvature-dependent (Hessian) damping intact. This allows the geometry of the loss landscape to naturally filter the disagreement: sharp, unstable directions are suppressed, while flat, stable directions are preserved.

Datasets

The authors evaluated their approach across various network scales and communication topologies using standard deep learning benchmarks:

  • CIFAR-10 and CIFAR-100: Used for image classification tasks, evaluated using Wide ResNet architectures (WRN16-8 and WRN28-10).
  • WMT14 (English-to-German): Used for neural machine translation using a Transformer-big model (~213M parameters) to validate how the concept translates to adaptive optimizers like Adam.

Key Insights and Ideas

  • Harnessing the "Bug" as a Feature: Rather than treating worker disagreement as a detrimental artifact of decentralized communication, it can be engineered into a structured, geometry-aware perturbation mechanism.
  • The Geometry Filter: Decaying the consensus factor doesn't just keep worker models apart randomly. Because the Hessian matrix still actively damps out variations along sharp directions, the remaining disagreement is forced to align with flat directions in the loss landscape.
  • Computational Efficiency: Explicit sharpness-aware methods (like SAM) require two forward/backward passes per iteration, doubling the computational cost. DSGD-AC achieves a similar regularizing effect for "free" by utilizing the spatial distribution of workers already inherent to decentralized architectures.

Key Results

  • Flatter Solutions: Empirical diagnostics confirmed that DSGD-AC successfully targets flatter regions, significantly reducing the dominant (top-1) Hessian eigenvalue compared to both standard DSGD and centralized synchronous SGD.
  • Superior Accuracy: On CIFAR-100 with a WRN28-10 model and an 8-worker ring topology, DSGD-AC achieved a test accuracy of 82.12%, markedly higher than standard DSGD (79.71%) and centralized synchronous SGD (80.00%).
Figure 2: Training performance and curvature for WRN16-8 on CIFAR-100 using 8 workers and the one- peer ring topology. DSGD-AC improves test accuracy and reaches a flatter local region than DSGD and synchronous SGD.
  • Preserved Training Efficiency: Decentralized methods in this study required only 0.78–0.85× the training time of synchronous centralized SGD due to overlapping communication and computation. DSGD-AC introduces negligible runtime overhead compared to standard DSGD.

Limitations and Future Work

  • Hessian Alignment Trade-off: The authors note that DSGD-AC actually exhibits a weaker normalized Hessian alignment than standard DSGD. Its superior performance relies entirely on its larger perturbation radius compensating for this weaker alignment. Designing a mixing operator that maintains both a large radius and strong alignment remains an open challenge.
  • Extension to Adaptive Optimizers: While the authors successfully combined adaptive consensus with Adam (creating DAdam-AC) for the WMT14 Transformer benchmark, the performance gains were more marginal than with SGD. Adaptive optimizers inherently reshape gradient noise and disrupt its natural alignment with the Hessian eigenspace, making this a complex area for future research.

Summary updated

Abstract

Decentralized training is often regarded as inferior to centralized training because the consensus errors between workers are thought to undermine convergence and generalization. This work challenges this view by introducing decentralized SGD with Adaptive Consensus (DSGD-AC), which uses a time-dependent scaling mechanism to maintain consensus errors throughout the training. We show that adaptive consensus changes the stationary variance of disagreement modes by balancing two effects: it preserves consensus-error magnitude through weaker graph damping while still allowing curvature-dependent damping to shape the disagreement directions. This balance can produce a stronger Hessian-weighted loss-envelope penalty around the deployed model, even when normalized Hessian alignment is weaker than in standard DSGD. Empirical results on image classification show that DSGD-AC reaches flatter solutions and higher test accuracy than standard DSGD and even centralized SGD. Together, these results support consensus errors as a useful implicit regularizer and open a new perspective on the design of decentralized learning algorithms.
BibTeX
@article{wang2026decentralized,
  author = {Zesen Wang and Mikael Johansson},
  title = {{Decentralized SGD with Controlled Disagreement Finds Flatter Minima}},
  journal = {Accepted by TMLR},
  year = {2026},
  url = {https://openreview.net/forum?id=DisRuxcT57},
}
Figure