Introduction
Federated Learning (FL) typically relies on a central server for model aggregation, creating a potential single point of failure and communication bottleneck. Decentralized Federated Learning (DFL) addresses this by allowing nodes to communicate directly with neighbors. However, DFL introduces its own challenges: high communication overhead, privacy risks, and the impact of "straggler" nodes. This paper introduces CEPS (Communication-Efficient and Private method for Sparse DFL), an algorithm that leverages one-bit compressive sensing to reduce data transmission by 90% while maintaining rigorous differential privacy guarantees.
The CEPS Framework
The CEPS algorithm addresses the efficiency-privacy-robustness trilemma through several integrated components:
- Sparse Optimization: By formulating the problem with an $L_0$-norm constraint, CEPS enforces model sparsity. This sparsity is critical for the subsequent compression step.
- One-Bit Compressive Sensing (1BCS): Instead of transmitting full 64-bit floating-point parameters, nodes send only the sign (1 bit) of their updates. The receiving nodes use a reconstruction algorithm (GPSP) to recover the sparse vectors.
- Inexact ADM Framework: The use of an Inexact Alternating Direction Method allows for computationally efficient, closed-form local updates.
- Differential Privacy (DP): A Gaussian mechanism adds calibrated noise to gradients, ensuring $(\epsilon, \delta)$-differential privacy.
- Asynchronous Communication: Nodes communicate with a subset of neighbors at periodic intervals, mitigating the impact of slower "straggler" devices.
Experimental Evaluation
The researchers evaluated CEPS on synthetic linear regression and real-world datasets including 'qot' (Qsar oral toxicity) and 'rls' (real-sim). These benchmarks tested the algorithm's ability to handle high-dimensional sparse models and its scalability across varying network sizes.
As shown in the figure above, CEPS maintains consistent convergence behavior as the number of nodes ($m$) increases from 32 to 128. The performance gap between the 1-bit compressed communication (DP-1BCS) and full-precision methods (DP-Perf) remains negligible, demonstrating the robustness of the 1BCS approach for large-scale decentralized networks.
Key Insights and Results
A primary finding is that decentralized consensus can be reached using only directional information (signs), provided the underlying model is sparse. While 1BCS reconstruction requires more local computation than simple averaging, this trade-off is highly beneficial in bandwidth-constrained environments, leading to a 90% reduction in total data transmission volume.
Furthermore, the integration of Differential Privacy shows that while high privacy levels (low $\epsilon$) introduce noise-induced fluctuations in the objective function, the decentralized nature of the algorithm allows the network to eventually "smooth out" these perturbations and reach convergence. This is illustrated in the figure above for $\epsilon=0.25$, where the objective function stabilizes despite initial noise-driven variance.
Conclusion
CEPS demonstrates that extreme communication compression via one-bit signals is a viable path for private, decentralized learning. While currently optimized for sparse convex problems like linear and logistic regression, future work may extend these 1-bit techniques to non-convex architectures such as deep neural networks, potentially addressing the computational overhead of local decoding on extremely low-power edge devices.
Summary updated