Accelerating Federated Learning via Sequential Training of Grouped Heterogeneous Clients

Andrea Silvi, Andrea Rizzardi, Debora Caldarola, Barbara Caputo, Marco CicconeIEEE Access 2024 · DOI

Abstract

Federated Learning (FL) faces significant challenges due to "client drift" caused by non-i.i.d. data distributions across participating devices. This paper introduces FedSeq, a novel framework that mitigates this drift by grouping heterogeneous clients into "superclients." By training clients sequentially within these groups, the model encounters a more representative data distribution before server-side aggregation. Experimental results demonstrate that FedSeq achieves up to 18x speedups in convergence and significantly higher accuracy compared to standard parallel training methods like FedAvg.

Introduction

Federated Learning (FL) enables collaborative model training without centralizing raw data, preserving user privacy. However, real-world data is often highly heterogeneous (non-i.i.d.), where local data distributions vary significantly between clients. Standard algorithms like FedAvg suffer from "client drift," where local updates diverge toward local optima, leading to slow convergence and suboptimal global performance.

Methodology

The proposed framework shifts the training paradigm from purely parallel updates to a hybrid sequential-parallel approach organized around the concept of "superclients."

Framework Overview

The FedSeq process is divided into three primary phases: comparing local data distributions, grouping clients into superclients, and executing sequential training.

The three-phase workflow of the FedSeq framework, from distribution comparison to sequential training variants.

Superclient Grouping

Unlike traditional clustering that groups similar items, FedSeq employs an "anti-clustering" strategy. The goal is to group clients with dissimilar data distributions into a single superclient. This ensures that the aggregate data within a superclient closely approximates the global data distribution. To achieve this in a privacy-preserving manner, the authors utilize Task2Vec embeddings, which characterize local tasks without requiring access to raw data or external public datasets.

Sequential Training (FedSeq)

Within each superclient, training proceeds sequentially. A model is passed through a chain of clients; each client performs local updates and passes the updated model to the next. Only the final model from the last client in the chain is transmitted back to the central server for global averaging.

System architecture showing superclients composed of heterogeneous devices training sequentially before communicating with the cloud server.

Advanced Variants

To address practical deployment challenges, two variants are introduced:

  • FedAsyncSeq: Mitigates the impact of "stragglers" (slow devices) by allowing superclients to operate asynchronously, passing models forward without waiting for global synchronization.
  • FedSeq2Par (Sequential-to-Parallel): A dynamic strategy that begins with sequential training to establish a robust global foundation and gradually transitions to parallel training. This prevents the model from overfitting to the specific ordering of clients within the chains.

Experimental Evaluation

The framework was evaluated across several benchmarks covering computer vision (CIFAR10, CIFAR100, FEMNIST) and Natural Language Processing (SHAKESPEARE, STACKOVERFLOW).

Key Results

FedSeq demonstrated substantial improvements over state-of-the-art baselines:

  • Convergence Speed: On the FEMNIST dataset, FedSeq achieved a 18x speedup to reach target accuracy. On StackOverflow, it achieved a 10x speedup.
  • Final Accuracy: In highly heterogeneous settings (CIFAR100, $\alpha=0$), FedSeq2Par reached 51.46% accuracy, outperforming FedAvg (42.68%) and FedDyn (50.35%).
  • Communication Efficiency: By reducing the frequency of server-client interactions through intra-group sequential passes, the framework significantly lowers communication overhead.

Discussion

Sequentiality as a Regularizer

The sequential chaining of updates acts as an implicit regularizer. It forces the model to find a parameter space that generalizes across multiple distinct local datasets before the server averages the results. This makes the resulting updates more stable and "digestible" for the global model.

Privacy Resilience

The authors found that sequential training does not compromise privacy. In fact, by producing more robust models, FedSeq increases resilience against "GAN recovery attacks," making it more difficult for adversaries to reconstruct private training data from model updates.

Conclusion and Future Work

FedSeq offers a powerful solution to the client drift problem in heterogeneous FL environments. While sequential training can introduce latency if chains are poorly managed, variants like FedAsyncSeq effectively mitigate these risks. Future research will explore optimized network topologies for client chains and the integration of differential privacy to further enhance data protection.

Summary updated

BibTeX
@article{silvi2024accelerating,
  author = {Andrea Silvi and Andrea Rizzardi and Debora Caldarola and Barbara Caputo and Marco Ciccone},
  title = {{Accelerating Federated Learning via Sequential Training of Grouped Heterogeneous Clients}},
  journal = {IEEE Access},
  year = {2024},
  doi = {10.1109/access.2024.3387453},
}
Figure