Introduction: The Challenge of Partial Participation
Federated Learning (FL) enables collaborative model training across decentralized devices while keeping data local. A significant practical hurdle is partial participation: due to connectivity or power constraints, only a small subset of clients can participate in any given communication round.
While primal-dual methods are theoretically robust for decomposing large-scale optimization problems, they suffer from a phenomenon known as Dual Drift in FL settings. When a client is inactive, its local dual variables—which track the gap between local and global consensus—become stale. Upon rejoining, these outdated variables introduce significant noise, leading to training instability or divergence.
The A-FedPD Method: Virtual Dual Updates
To address this, the authors introduce Aligned Federated Primal Dual (A-FedPD). The core mechanism is the virtual dual update. Unlike traditional methods where only active clients update their dual variables, A-FedPD empowers the server to perform "virtual" updates for inactive clients.
By using the global model update as a proxy for the missing local updates, the server keeps the dual variables of all clients synchronized with the global state. This "alignment" prevents the sudden shocks caused by stale variables, effectively simulating a full-participation environment. Theoretically, A-FedPD achieves an $O(1/T)$ convergence rate for non-convex objectives, ensuring both stability and speed.
Experimental Results
The researchers evaluated A-FedPD on CIFAR-10 and CIFAR-100 datasets with varying degrees of data heterogeneity (modeled via Dirichlet distributions).
Performance and Scalability
A-FedPD demonstrates remarkable robustness to low participation rates. As shown in the figure below, the training loss and test accuracy remain stable even as the percentage of participating clients decreases, a scenario where many existing primal-dual methods struggle with oscillation.
Comparison with State-of-the-Art
In head-to-head comparisons, A-FedPD consistently outperforms established baselines like FedAvg, SCAFFOLD, and FedDyn. Notably, on the highly heterogeneous CIFAR-100 (Dir-0.1) task, A-FedPD combined with Sharpness-Aware Minimization (SAM) achieved a top accuracy of 53.15%, significantly higher than FedAvg's 38.37%.
The method also offers substantial communication efficiency, reaching target accuracies up to 6x faster than standard approaches on complex datasets like CIFAR-100.
Conclusion and Future Work
A-FedPD solves the critical dual drift problem in federated primal-dual learning through a simple yet effective alignment strategy. By maintaining virtual dual variables for inactive clients, it achieves superior stability and convergence speeds in realistic, low-participation FL environments. Future work will focus on reducing server-side storage requirements for these dual variables to support massive-scale deployments with millions of edge devices.
Summary updated