TL;DR
This paper provides a much-needed experimental comparison of three Federated Learning (FL) architectures: Centralized (CFL), Decentralized (DFL), and Semi-Decentralized (SDFL). By simulating these architectures on the MNIST dataset with an MLP classifier, the authors reveal crucial performance trade-offs, showing that DFL often offers the best accuracy and resource efficiency, while CFL excels in faster convergence.
What Problem Does This Paper Address?
The explosion of data from IoT devices presents a significant challenge for traditional centralized machine learning. Federated Learning (FL) offers a privacy-preserving solution by training models collaboratively on edge devices without sharing raw data. However, FL can be implemented in various architectures: Centralized (CFL), Decentralized (DFL), and Semi-Decentralized (SDFL). While research has explored these architectures individually, there's a significant gap in experimentally comparing their performance across key metrics. This lack of comparative analysis makes it difficult for practitioners to choose the most suitable architecture for their specific needs, especially considering potential trade-offs in accuracy, communication costs, and resource usage.
Methods
The authors employ a simulation-based approach using the FedStellar simulator, a platform that supports all three FL architectures and allows for the monitoring of various Key Performance Indicators (KPIs). The experimental setup involves:
- Simulating FL processes: The core FL training loop, including local training, model updates, and aggregation, is simulated.
- Implementing KPIs: The study focuses on measuring several crucial KPIs, including:
- Model Performance: Accuracy, Precision, Recall, and F-score.
- Resource Usage: CPU usage percentage.
- Communication Cost: Bytes exchanged in the network.
- Scalability Analysis: Experiments are conducted with varying numbers of participants (3, 4, 6, and 8) to assess how each architecture performs under different scales.
- Trade-off Identification: The results are analyzed to highlight the inherent trade-offs between these KPIs for each architecture.
Datasets
- MNIST: A widely used dataset of handwritten digits, consisting of 60,000 training images and 10,000 testing images, categorized into 10 classes. The dataset is partitioned among participants to simulate a distributed learning environment.
Key Insights and Ideas
- DFL as a Performance Leader: Decentralized Federated Learning (DFL) consistently demonstrates the highest accuracy and lowest loss across various participant numbers, often outperforming both CFL and SDFL.
- CFL's Speed Advantage: Despite lower overall performance, Centralized Federated Learning (CFL) exhibits the fastest convergence to a stable performance level, making it suitable when speed is paramount.
- SDFL: A Balanced Approach: Semi-Decentralized Federated Learning (SDFL) offers a compelling middle ground, achieving performance close to DFL while often converging faster than DFL, especially with fewer participants.
- Scalability Resilience of DFL/SDFL: Unlike CFL, where performance tends to degrade as the number of participants increases (due to smaller local datasets per participant), DFL and SDFL maintain stable performance. This resilience is attributed to their decentralized nature, where each node works with its local data.
- Communication Efficiency: DFL generally shows lower communication costs compared to CFL, which is expected given the absence of a central aggregator. SDFL's communication costs are comparable to DFL.
- Resource Consumption: CFL is identified as the most resource-intensive architecture, particularly in terms of network communication, due to its reliance on a central server. DFL and SDFL are more lightweight in this regard.
Key Results
- With 3 participants: DFL achieved the highest accuracy (97.84%) and lowest loss (0.0729), while CFL was faster to converge but less performant (94.45% accuracy, 0.1617 loss) and more resource-intensive.
- As participants increase (4, 6, 8): DFL and SDFL consistently maintained high accuracy (above 96%), while CFL's accuracy declined (down to 89.83% with 8 participants).
- Convergence Time: CFL consistently converged the fastest. DFL and SDFL generally took longer to converge, though SDFL sometimes converged faster than DFL for smaller participant counts.
- Communication Cost: DFL was the least expensive in terms of communication, with SDFL being comparable. CFL was the most expensive.
- CPU Usage: DFL and SDFL generally showed lower CPU usage compared to CFL.
Limitations and Future Work
- Limited KPIs: The study acknowledges that FedStellar currently permits the measurement of only a subset of relevant KPIs. Future work will involve expanding the range of measured KPIs to provide a more comprehensive analysis.
- Simulator Constraints: The experiments were initially limited to 3 participants due to computational constraints, though scalability was then tested with up to 8 participants.
- Future Research Directions: The authors plan to:
- Cover a wider range of KPIs and explore further trade-offs.
- Extend the FedStellar platform's code.
- Develop a decision support tool to help users select the most appropriate FL architecture and platform based on their specific needs.
Summary updated