Abstract
Federated Learning (FL) faces significant challenges due to "client drift" caused by non-i.i.d. data distributions across participating devices. This paper introduces FedSeq, a novel framework that mitigates this drift by grouping heterogeneous clients into "superclients." By training clients sequentially within these groups, the model encounters a more representative data distribution before server-side aggregation. Experimental results demonstrate that FedSeq achieves up to 18x speedups in convergence and significantly higher accuracy compared to standard parallel training methods like FedAvg.
Introduction
Federated Learning (FL) enables collaborative model training without centralizing raw data, preserving user privacy. However, real-world data is often highly heterogeneous (non-i.i.d.), where local data distributions vary significantly between clients. Standard algorithms like FedAvg suffer from "client drift," where local updates diverge toward local optima, leading to slow convergence and suboptimal global performance.
Methodology
The proposed framework shifts the training paradigm from purely parallel updates to a hybrid sequential-parallel approach organized around the concept of "superclients."
Framework Overview
The FedSeq process is divided into three primary phases: comparing local data distributions, grouping clients into superclients, and executing sequential training.
Superclient Grouping
Unlike traditional clustering that groups similar items, FedSeq employs an "anti-clustering" strategy. The goal is to group clients with dissimilar data distributions into a single superclient. This ensures that the aggregate data within a superclient closely approximates the global data distribution. To achieve this in a privacy-preserving manner, the authors utilize Task2Vec embeddings, which characterize local tasks without requiring access to raw data or external public datasets.
Sequential Training (FedSeq)
Within each superclient, training proceeds sequentially. A model is passed through a chain of clients; each client performs local updates and passes the updated model to the next. Only the final model from the last client in the chain is transmitted back to the central server for global averaging.
Advanced Variants
To address practical deployment challenges, two variants are introduced:
- FedAsyncSeq: Mitigates the impact of "stragglers" (slow devices) by allowing superclients to operate asynchronously, passing models forward without waiting for global synchronization.
- FedSeq2Par (Sequential-to-Parallel): A dynamic strategy that begins with sequential training to establish a robust global foundation and gradually transitions to parallel training. This prevents the model from overfitting to the specific ordering of clients within the chains.
Experimental Evaluation
The framework was evaluated across several benchmarks covering computer vision (CIFAR10, CIFAR100, FEMNIST) and Natural Language Processing (SHAKESPEARE, STACKOVERFLOW).
Key Results
FedSeq demonstrated substantial improvements over state-of-the-art baselines:
- Convergence Speed: On the FEMNIST dataset, FedSeq achieved a 18x speedup to reach target accuracy. On StackOverflow, it achieved a 10x speedup.
- Final Accuracy: In highly heterogeneous settings (CIFAR100, $\alpha=0$), FedSeq2Par reached 51.46% accuracy, outperforming FedAvg (42.68%) and FedDyn (50.35%).
- Communication Efficiency: By reducing the frequency of server-client interactions through intra-group sequential passes, the framework significantly lowers communication overhead.
Discussion
Sequentiality as a Regularizer
The sequential chaining of updates acts as an implicit regularizer. It forces the model to find a parameter space that generalizes across multiple distinct local datasets before the server averages the results. This makes the resulting updates more stable and "digestible" for the global model.
Privacy Resilience
The authors found that sequential training does not compromise privacy. In fact, by producing more robust models, FedSeq increases resilience against "GAN recovery attacks," making it more difficult for adversaries to reconstruct private training data from model updates.
Conclusion and Future Work
FedSeq offers a powerful solution to the client drift problem in heterogeneous FL environments. While sequential training can introduce latency if chains are poorly managed, variants like FedAsyncSeq effectively mitigate these risks. Future research will explore optimized network topologies for client chains and the integration of differential privacy to further enhance data protection.
Summary updated