A Distributed and Elastic Aggregation Service for Scalable Federated Learning Systems

Ahmad Khan, Yuze Li, Ali Anwar, Yue Cheng, Thang Hoang, Nathalie Baracaldo, Ali Raza ButtarXiv 2022 · arXiv · OpenReview

Abstract

Federated Learning (FL) has promised a new approach to resolve the challenges in machine learning by bringing computation to the data. However, state-of-the-art systems often rely on a single-node aggregator that is insufficient for training large device populations or complex models. As model sizes and client counts increase, single-node aggregators face significant memory, computation, and communication bottlenecks. We propose a new aggregation service that classifies workloads and adaptively selects the best solution based on model size and client volume. Our system provides a fault-tolerant, robust, and efficient solution utilizing existing parallel and distributed frameworks. Through extensive evaluation, we demonstrate the shortcomings of static approaches and show how our elastic service outperforms current frameworks.

Introduction: The Aggregator Bottleneck

Federated Learning allows devices to train machine learning models locally, preserving privacy by keeping raw data on-device. The central challenge lies in the "aggregator"—the server responsible for fusing local updates into a global model. Most existing frameworks assume the aggregator resides in a cloud environment with virtually infinite resources.

However, deploying FL at the "Edge" (near IoT devices) introduces severe constraints. Edge data centers have limited CPU, memory, and bandwidth. Static aggregation methods—those that use a fixed resource allocation—fail in these environments; they are either too slow for large client pools or prohibitively expensive for small ones. This paper addresses the need for a resource-aware, elastic aggregation service that scales dynamically with the workload.

An Adaptive, Multi-Tiered Framework

The core innovation is a hybrid framework that utilizes a Q-learning-based Reinforcement Learning (RL) agent to select the optimal aggregation strategy for each training round. The agent monitors state variables such as the number of clients, model size, and available system resources to choose between three distinct methodologies:

  1. Numba-based (Multi-core): This tier is optimized for single-node performance. By using Numba to parallelize Python loops, the system achieves high efficiency for smaller workloads without the overhead of distributed systems.
  2. Spark-based (Multi-node): For large models and high client participation, the system leverages Apache Spark. This allows for horizontal scaling, distributing heavy I/O and decompression tasks across a cluster.
  3. Serverless (AWS Lambda): Utilizing a "tree-reduce" approach, this tier triggers massive numbers of short-lived functions. It is ideal for high-concurrency scenarios with many small updates, though it can become costly for very large models.

Experimental Evaluation

The system was evaluated using a variety of model architectures, including custom CNNs (from 4.6 MB to 956 MB) and standard benchmarks like ResNet50 and VGG16. A custom client emulator simulated up to 1 million clients across global AWS regions.

Performance Gains with Numba

For smaller client counts, the Numba-optimized approach significantly outperforms vanilla Python implementations. By parallelizing the fusion algorithm, the system reduces the CPU bottleneck often found at the aggregator.

Comparison of Vanilla vs. Numba-optimized aggregation for ResNet50 models.

Scaling to the Massive Edge

As the number of sampled clients grows into the tens of thousands, the Spark-based distributed approach becomes essential. The evaluation showed that while traditional methods might crash under edge resource limits, the proposed system maintains a linear scaling trend.

Scalability of FedAvg and IterAvg aggregation methods up to 100,000 clients.

Key Insights

  • CPU as the Primary Bottleneck: While network latency is a common concern in FL, the complexity of fusion algorithms like FedAvg creates a significant CPU bottleneck. Smart software scheduling and parallelization (like Numba) are often more cost-effective than specialized GPU hardware.
  • The Decompression Tax: Model compression saves bandwidth but requires the aggregator to perform intensive decompression. The Spark-based method mitigates this by parallelizing decompression across worker nodes.
  • No "One Size Fits All": The "best" method changes drastically based on the model. For small models, the adaptive method reduced latency by 88% simply by avoiding the overhead of Spark or Serverless functions.

Conclusion and Future Work

The proposed elastic aggregation service achieves up to an 8x improvement in time efficiency and a 2x reduction in operational costs compared to static approaches. By dynamically matching the aggregation strategy to the workload, the system enables scalable FL in resource-constrained edge environments.

Future research will focus on integrating non-synchronous fusion algorithms and exploring the use of specialized edge AI accelerators. Additionally, refining the RL agent to better handle "straggler" clients—devices with unpredictable reporting times—remains a priority for improving real-world robustness.

Summary updated

Abstract

Federated Learning has promised a new approach to resolve the challenges in machine learning by bringing computation to the data. The popularity of the approach has led to rapid progress in the algorithmic aspects and the emergence of systems capable of simulating Federated Learning. State of art systems in Federated Learning support a single node aggregator that is insufficient to train a large corpus of devices or train larger-sized models. As the model size or the number of devices increase the single node aggregator incurs memory and computation burden while performing fusion tasks. It also faces communication bottlenecks when a large number of model updates are sent to a single node. We classify the workload for the aggregator into categories and propose a new aggregation service for handling each load. Our aggregation service is based on a holistic approach that chooses the best solution depending on the model update size and the number of clients. Our system provides a fault-tolerant, robust and efficient aggregation solution utilizing existing parallel and distributed frameworks. Through evaluation, we show the shortcomings of the state of art approaches and how a single solution is not suitable for all aggregation requirements. We also provide a comparison of current frameworks with our system through extensive experiments.
BibTeX
@misc{khan2022distributed,
  author = {Ahmad Khan and Yuze Li and Ali Anwar and Yue Cheng and Thang Hoang and Nathalie Baracaldo and Ali Raza Butt},
  title = {{A Distributed and Elastic Aggregation Service for Scalable Federated Learning Systems}},
  year = {2022},
  eprint = {2204.07767},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2204.07767},
}
Figure