FedDrop: Trajectory-weighted Dropout for Efficient Federated Learning

Dongping Liao, Xitong Gao, Yiren Zhao, Hao Dai, Li Li, Kafeng Wang, Kejiang Ye, Yang Wang, Cheng-zhong XuICLR 2022 · OpenReview

Abstract

Federated learning (FL) enables edge clients to train collaboratively while preserving individual's data privacy. As clients do not inherently share identical data distributions, they may disagree in the direction of parameter updates, resulting in high compute and communication costs in comparison to centralized learning. Recent advances in FL focus on reducing data transmission during training; yet they neglected the increase of computational cost that dwarfs the merit of reduced communication. To this end, we propose FedDrop, which introduces channel-wise weighted dropout layers between convolutions to accelerate training while minimizing their impact on convergence. Empirical results show that FedDrop can drastically reduce the amount of FLOPs required for training with a small increase in communication, and push the Pareto frontier of communication/computation trade-off further than competing FL algorithms.
BibTeX
@inproceedings{liao2022feddrop,
  author = {Dongping Liao and Xitong Gao and Yiren Zhao and Hao Dai and Li Li and Kafeng Wang and Kejiang Ye and Yang Wang and Cheng-zhong Xu},
  title = {{FedDrop: Trajectory-weighted Dropout for Efficient Federated Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=obi9EkyVeED},
}

Loading related content...

Figure