On Improved Distributed Random Reshuffling over Networks

Pranay Sharma, Jiarui Li, Gauri JoshiICASSP 2024 · OpenReview

Abstract

In this paper, we consider a distributed optimization problem. A network of n agents, each with its own local loss function, aims to collaboratively minimize the global average loss. We prove improved convergence results for two recently proposed random reshuffling (RR) based algorithms, D-RR and GT-RR, for smooth strongly-convex and nonconvex problems, respectively. In particular, we prove an additional speedup with increasing n in both cases. Our experiments show that these methods can provide further communication savings by carrying multiple gradient steps between successive communications while also outperforming decentralized SGD. Our experiments also reveal a gap in the theoretical understanding of these methods in the nonconvex case.
BibTeX
@inproceedings{sharma2024improved,
  author = {Pranay Sharma and Jiarui Li and Gauri Joshi},
  title = {{On Improved Distributed Random Reshuffling over Networks}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=BOA1goJsJJ},
}

Loading related content...

Figure