Decentralized Nonconvex Composite Federated Learning with Gradient Tracking and Momentum

Yuan Zhou, Xinli Shi, Xuelong Li, Jiachen Zhong, Guanghui Wen, Jinde Cao

Abstract

Decentralized Federated Learning (DFL) enables collaborative model training without relying on a central server. When local objectives are nonconvex and coupled with nonsmooth weakly convex regularization, DFL gives rise to a challenging decentralized nonconvex composite optimization problem involving data heterogeneity, stochastic-gradient noise, and consensus error. We propose DEPOSITUM, a decentralized composite optimization algorithm for this problem. DEPOSITUM maintains momentum-filtered stochastic gradient estimates via a tracking mechanism, which accommodates both Polyak and Nesterov momentum. It further allows multiple local updates between communication rounds to improve communication efficiency. Theoretical analysis demonstrates that it achieves an expected $ε$-stationary point with an iteration complexity of $\mathcal{O}(1/ε^2)$ without imposing bounded gradient heterogeneity or mean-squared smoothness assumptions. With an appropriate stepsize and momentum schedule, the averaged stationarity measure further achieves a rate of \(\mathcal{O}(1/\sqrt{nT})\) after a network-dependent transient, using a mini-batch size independent of $T$. Experiments on different benchmark datasets validate the effectiveness of DEPOSITUM and demonstrate competitive performance against representative federated composite optimization methods.
BibTeX
@misc{zhou2025decentralized,
  author = {Yuan Zhou and Xinli Shi and Xuelong Li and Jiachen Zhong and Guanghui Wen and Jinde Cao},
  title = {{Decentralized Nonconvex Composite Federated Learning with Gradient Tracking and Momentum}},
  year = {2025},
  eprint = {2504.12742},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.12742},
}

Loading related content...

Figure