An Empirical Study of Efficiency and Privacy of Federated Learning Algorithms

Sofia Zahri, Hajar Bennouri, Ahmed M. AbdelmoniemarXiv 2023 · OpenReview

TL;DR

This paper empirically investigates the efficiency and privacy trade-offs of Federated Learning (FL) algorithms, specifically FedAvg and FedProx, in the context of Internet of Things (IoT) networks. It demonstrates that FedProx offers improved convergence, especially under heterogeneous network conditions, and explores the impact of Differential Privacy (DP) on model accuracy, highlighting a clear privacy-accuracy trade-off.

What Problem Does This Paper Address?

The explosion of IoT devices generates vast amounts of heterogeneous data, posing significant challenges for traditional centralized machine learning approaches. These challenges include bandwidth limitations, latency, and, crucially, privacy concerns as sensitive user data is collected and processed centrally. Federated Learning (FL) emerges as a promising solution by enabling model training directly on user devices, thus preserving data privacy. However, FL itself faces hurdles like system heterogeneity (varying device capabilities) and statistical heterogeneity (non-IID data distributions), which can impact model efficiency and convergence. This paper aims to empirically evaluate and compare two key FL algorithms, FedAvg and FedProx, in terms of their efficiency and then investigate how Differential Privacy (DP) affects their performance and privacy guarantees in realistic IoT scenarios.

Methods

The study employs a two-pronged experimental approach:

  1. Efficiency Evaluation with Flower: The researchers utilize the Flower framework, a popular tool for federated learning simulations, to compare the performance of FedAvg and FedProx. They train deep neural networks on common datasets and analyze accuracy and loss metrics. This part focuses on understanding how FedProx, with its "proximal term," handles system and statistical heterogeneity compared to the standard FedAvg.
  2. Privacy Evaluation with Opacus: To assess privacy preservation, they integrate Differential Privacy (DP) using the Opacus library in PyTorch. This involves adding noise to model gradients during training. The experiments then compare the performance of DP-enhanced FedAvg (DP_FedAvg) and DP-enhanced FedProx (DP_FedProx) against their non-private counterparts, analyzing the trade-off between enhanced privacy and model accuracy.

Datasets

The paper uses the MNIST dataset for its experiments. The dataset is partitioned in a "pathological split" where each client is assigned examples from only two out of ten classes, simulating a non-IID data distribution characteristic of real-world IoT environments.

Key Insights and Ideas

  • FedProx Shines in Heterogeneity: FedProx demonstrates superior convergence and stability compared to FedAvg, particularly when dealing with significant system heterogeneity (e.g., a high percentage of "straggler" clients that are slow or drop out) and non-IID data. The proximal term in FedProx effectively mitigates the negative impact of these challenging conditions.
  • The Inevitable Privacy-Accuracy Trade-off: Implementing Differential Privacy, while significantly enhancing data privacy by adding noise to gradients, inevitably leads to a reduction in model accuracy. This is a fundamental trade-off that needs careful consideration when deploying privacy-preserving FL.
  • Flower and Opacus: Practical Tools for FL Research: The study highlights the utility of frameworks like Flower for simulating FL scenarios and Opacus for easily integrating DP into PyTorch models, making empirical research in this area more accessible.

Key Results

  • FedProx vs. FedAvg: Figures 3-5 show that FedProx consistently achieves better or comparable accuracy and loss compared to FedAvg, especially under high straggler percentages (90%). The effectiveness of FedProx is shown to be influenced by the proximal term ($\mu$), with higher values (e.g., $\mu=2.0$) leading to more stable convergence.
  • DP Impact: Figures 6 and 7 illustrate the impact of DP. For FedAvg, DP_FedAvg shows a noticeable drop in accuracy compared to FedAvg. For FedProx, DP_FedProx also exhibits lower accuracy than its non-DP counterpart, reaching approximately 0.8353 accuracy in the 20th round compared to 0.9146 for FedProx without DP.

Limitations and Future Work

  • Limited Scope of Datasets: The experiments primarily focus on the MNIST dataset. While it's used to simulate non-IID conditions, testing on more complex and diverse datasets would provide broader insights.
  • Focus on Specific Algorithms: The study concentrates on FedAvg and FedProx. Future work could explore other advanced FL algorithms and their privacy-efficiency trade-offs.
  • Future Directions: The authors suggest further research into optimizing DP mechanisms for FL, exploring more sophisticated privacy-preserving techniques, and investigating the impact of various system and data heterogeneity levels on FL performance. They also emphasize the need for continued research into efficient FL for IoT networks.

Summary updated

Abstract

In today's world, the rapid expansion of IoT networks and the proliferation of smart devices in our daily lives, have resulted in the generation of substantial amounts of heterogeneous data. These data forms a stream which requires special handling. To handle this data effectively, advanced data processing technologies are necessary to guarantee the preservation of both privacy and efficiency. Federated learning emerged as a distributed learning method that trains models locally and aggregates them on a server to preserve data privacy. This paper showcases two illustrative scenarios that highlight the potential of federated learning (FL) as a key to delivering efficient and privacy-preserving machine learning within IoT networks. We first give the mathematical foundations for key aggregation algorithms in federated learning, i.e., FedAvg and FedProx. Then, we conduct simulations, using Flower Framework, to show the \textit{efficiency} of these algorithms by training deep neural networks on common datasets and show a comparison between the accuracy and loss metrics of FedAvg and FedProx. Then, we present the results highlighting the trade-off between maintaining privacy versus accuracy via simulations - involving the implementation of the differential privacy (DP) method - in Pytorch and Opacus ML frameworks on common FL datasets and data distributions for both FedAvg and FedProx strategies.
BibTeX
@misc{zahri2023empirical,
  author = {Sofia Zahri and Hajar Bennouri and Ahmed M. Abdelmoniem},
  title = {{An Empirical Study of Efficiency and Privacy of Federated Learning Algorithms}},
  year = {2023},
  eprint = {2312.15375},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.15375},
}
Figure