Decentralized Bilevel Optimization for Personalized Client Learning

Songtao Lu, Xiaodong Cui, Mark S. Squillante, Brian Kingsbury, Lior HoreshICASSP 2022 · OpenReview

Abstract

Decentralized optimization with multiple networked clients/learners has advanced machine learning significantly over the past few years. When data distributions at different nodes/locations are heterogeneous, consensus-based decentralized algorithms ignore distinctive features of local data samples. In this paper, we propose a decentralized client adaptation strategy for personalized learning by taking local client data structures into account. It turns out that optimizing the model parameters can be formulated as a decentralized bilevel programming problem. Motivated by this application, we propose a stochastic primal-dual framework for solving decentralized bilevel nonconvex problems and show that the devised algorithm achieves the Karush–Kuhn–Tucker (KKT) points for this class of problems at a rate of $\mathcal{O}(1/\sqrt {nT} )$, where n denotes the number of total learners and T the total number of iterations. Multiple experiments show the superiority of our proposed method compared to state-of-the-art methods in terms of both training speed and testing accuracy for decentralized learning problems on real datasets.
BibTeX
@inproceedings{lu2022decentralized,
  author = {Songtao Lu and Xiaodong Cui and Mark S. Squillante and Brian Kingsbury and Lior Horesh},
  title = {{Decentralized Bilevel Optimization for Personalized Client Learning}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=7ZY-7dveryY},
}

Loading related content...

Figure