Towards cost-effective and resource-aware aggregation at Edge for Federated Learning

Ahmad Faraz Khan, Yuze Li, Xinran Wang, Sabaat Haroon, Haider Ali, Yue Cheng, Ali Raza Butt, Ali AnwarIEEE Big Data 2022 · DOI · OpenReview

TL;DR

This paper tackles the challenge of efficient and cost-effective model aggregation in Federated Learning (FL) for resource-constrained edge environments. They introduce the first adaptive FL aggregator that dynamically selects the best aggregation strategy (Numba, Spark, or Serverless) based on workload and user preferences, significantly improving scalability, time efficiency, and reducing costs compared to static cloud-based methods.

What Problem Does This Paper Address?

Federated Learning (FL) is a powerful technique for training machine learning models on decentralized data without compromising privacy. A key component of FL is the aggregator server, which collects model updates from clients and combines them into a global model. While traditional FL often relies on powerful cloud servers, the rise of Edge and IoT applications necessitates performing aggregation closer to the data source, in resource-limited edge data centers. Existing cloud-centric aggregation solutions are ill-suited for these edge environments due to their high resource demands, leading to poor scalability, increased latency, and prohibitive costs. This paper addresses the critical need for FL aggregation strategies that are specifically designed for the constraints of edge data centers, balancing efficiency, scalability, and cost-effectiveness.

Methods

The core contribution of this paper is the development of an adaptive FL aggregator for edge environments. This adaptive aggregator intelligently selects from three distinct aggregation methodologies:

  • Numba-based Multi-core Aggregation: This method leverages Numba for parallel processing on a single node, optimizing for faster aggregation of smaller models and lower client participation. It's an enhancement of existing multi-core approaches.
  • Spark-based Multi-node Aggregation: Utilizing Apache Spark, this approach distributes aggregation tasks across multiple nodes, making it suitable for handling larger models and a higher number of clients. It leverages scalable storage like HDFS.
  • Serverless-based Multi-node Aggregation: This method employs serverless functions (AWS Lambda) for aggregation, offering a cost-effective solution for lighter workloads and fluctuating client participation. It utilizes a tree-reduce principle for efficient distribution.

The adaptive aggregator uses a Q-learning-based Reinforcement Learning (RL) agent to dynamically choose the optimal aggregation method. The RL agent learns from state variables such as task-specific information (workload) and system resource data (CPU, memory, available containers/functions). It aims to optimize a trade-off between completion time and cost, guided by user preferences. The system seamlessly switches between methodologies without disrupting clients, ensuring a consistent I/O channel.

Datasets

The paper does not explicitly mention specific datasets used for training FL models. However, the evaluation uses various Convolutional Neural Network (CNN) models of different sizes (e.g., CNN4.6, CNN73, CNN179, CNN239, CNN478, CNN717, CNN956) and established architectures like Resnet50 and VGG16. These models are used to simulate realistic FL workloads and evaluate the performance of the proposed aggregation methods. The paper also introduces a client emulator that simulates client behavior and interactions with the aggregator, which can be considered a benchmark for evaluating FL aggregation systems.

Key Insights and Ideas

  • No One-Size-Fits-All: The research strongly emphasizes that no single aggregation method is optimal for all edge FL scenarios. The best approach depends heavily on model size, the number of participating clients, and user-defined priorities (time vs. cost).
  • Serverless Cost Surprises: Counter-intuitively, serverless solutions, while often lauded for cost-effectiveness, can become surprisingly expensive for specific workloads, especially when dealing with very large models or high cold-start container requirements. This highlights the need for careful selection.
  • Spark's Scalability Advantage: For larger models and a significant number of clients, Spark-based multi-node aggregation demonstrates superior scalability and efficiency due to its distributed processing capabilities.
  • Numba for Lighter Loads: Numba-based multi-core aggregation proves to be efficient and cost-effective for lighter models and scenarios with fewer clients, where the overhead of distributed systems might be unnecessary.
  • Adaptive Control is Crucial: The adaptive aggregator, driven by RL, is essential for navigating the complex trade-offs in edge FL, ensuring that the system can dynamically adjust to changing demands and optimize for user-defined goals.

Key Results

The adaptive FL aggregator demonstrated significant improvements over static cloud-based methodologies:

  • Scalability: Improved by up to 4x.
  • Time Efficiency: Improved by up to 8x.
  • Cost Reduction: Reduced costs by more than 2x.
  • For specific workloads like CNN956, the adaptive method reduced latency by 63.51% and saved $0.2 per run compared to a static Spark method.
  • The Spark-based method showed a 3x increase in scalability compared to Vanilla for FedAvg.
  • Serverless improved aggregation efficiency by up to 87% for both FedAvg and IterAvg compared to Spark.

Limitations and Future Work

  • Model Size and Complexity: While the paper explores various model sizes, the performance of the adaptive aggregator might still be influenced by extreme model complexities not fully covered.
  • Network Variability: Although the emulator simulates dropouts, the impact of highly variable and unreliable network conditions in real-world edge deployments could be further investigated.
  • Multi-tenancy Isolation: While multi-tenancy was tested, a deeper dive into resource management and isolation for multiple tenants within the adaptive aggregator could be a future direction.
  • Exploration of Other Fusion Algorithms: The paper focuses on FedAvg and IterAvg. Exploring the adaptive aggregator's performance with other advanced fusion algorithms could be beneficial.
  • Real-world Deployment: The experiments were conducted using simulations and emulators. Deploying and evaluating the adaptive aggregator in real-world edge environments would provide further validation.

Summary updated

Abstract

Federated Learning (FL) is a machine learning approach that addresses privacy and data transfer costs by computing data at the source. It’s particularly popular for Edge and IoT applications where the aggregator server of FL is in resource-capped edge data centers for reducing communication costs. Existing cloud-based aggregator solutions are resource-inefficient and expensive at the Edge, leading to low scalability and high latency. To address these challenges, this study compares prior and new aggregation methodologies under the changing demands of IoT and Edge applications. This work is the first to propose an adaptive FL aggregator at the Edge, enabling users to manage the cost and efficiency trade-off. An extensive comparative analysis demonstrates that the design improves scalability by up to 4$\times$, time efficiency by 8$\times$, and reduces costs by more than 2$\times$ compared to extant cloud-based static methodologies.
BibTeX
@inproceedings{khan2022towards,
  author = {Ahmad Faraz Khan and Yuze Li and Xinran Wang and Sabaat Haroon and Haider Ali and Yue Cheng and Ali Raza Butt and Ali Anwar},
  title = {{Towards cost-effective and resource-aware aggregation at Edge for Federated Learning}},
  booktitle = {IEEE Big Data},
  year = {2022},
  doi = {10.48550/arxiv.2204.07767},
}

Loading related content...

Figure