Communication-Efficient Decentralized Federated Learning via One-Bit Compressive Sensing

Shenglong Zhou, Kaidi Xu, Geoffrey Ye Li

Introduction

Federated Learning (FL) typically relies on a central server for model aggregation, creating a potential single point of failure and communication bottleneck. Decentralized Federated Learning (DFL) addresses this by allowing nodes to communicate directly with neighbors. However, DFL introduces its own challenges: high communication overhead, privacy risks, and the impact of "straggler" nodes. This paper introduces CEPS (Communication-Efficient and Private method for Sparse DFL), an algorithm that leverages one-bit compressive sensing to reduce data transmission by 90% while maintaining rigorous differential privacy guarantees.

The CEPS Framework

The CEPS algorithm addresses the efficiency-privacy-robustness trilemma through several integrated components:

  1. Sparse Optimization: By formulating the problem with an $L_0$-norm constraint, CEPS enforces model sparsity. This sparsity is critical for the subsequent compression step.
  2. One-Bit Compressive Sensing (1BCS): Instead of transmitting full 64-bit floating-point parameters, nodes send only the sign (1 bit) of their updates. The receiving nodes use a reconstruction algorithm (GPSP) to recover the sparse vectors.
  3. Inexact ADM Framework: The use of an Inexact Alternating Direction Method allows for computationally efficient, closed-form local updates.
  4. Differential Privacy (DP): A Gaussian mechanism adds calibrated noise to gradients, ensuring $(\epsilon, \delta)$-differential privacy.
  5. Asynchronous Communication: Nodes communicate with a subset of neighbors at periodic intervals, mitigating the impact of slower "straggler" devices.

Experimental Evaluation

The researchers evaluated CEPS on synthetic linear regression and real-world datasets including 'qot' (Qsar oral toxicity) and 'rls' (real-sim). These benchmarks tested the algorithm's ability to handle high-dimensional sparse models and its scalability across varying network sizes.

Scalability across network sizes

As shown in the figure above, CEPS maintains consistent convergence behavior as the number of nodes ($m$) increases from 32 to 128. The performance gap between the 1-bit compressed communication (DP-1BCS) and full-precision methods (DP-Perf) remains negligible, demonstrating the robustness of the 1BCS approach for large-scale decentralized networks.

Key Insights and Results

A primary finding is that decentralized consensus can be reached using only directional information (signs), provided the underlying model is sparse. While 1BCS reconstruction requires more local computation than simple averaging, this trade-off is highly beneficial in bandwidth-constrained environments, leading to a 90% reduction in total data transmission volume.

Convergence under high privacy

Furthermore, the integration of Differential Privacy shows that while high privacy levels (low $\epsilon$) introduce noise-induced fluctuations in the objective function, the decentralized nature of the algorithm allows the network to eventually "smooth out" these perturbations and reach convergence. This is illustrated in the figure above for $\epsilon=0.25$, where the objective function stabilizes despite initial noise-driven variance.

Conclusion

CEPS demonstrates that extreme communication compression via one-bit signals is a viable path for private, decentralized learning. While currently optimized for sparse convex problems like linear and logistic regression, future work may extend these 1-bit techniques to non-convex architectures such as deep neural networks, potentially addressing the computational overhead of local decoding on extremely low-power edge devices.

Summary updated

BibTeX
@inproceedings{zhou2024communication,
  author = {Shenglong Zhou and Kaidi Xu and Geoffrey Ye Li},
  title = {{Communication-Efficient Decentralized Federated Learning via One-Bit Compressive Sensing}},
  booktitle = {VTC Spring},
  year = {2024},
  doi = {10.1109/vtc2024-spring62846.2024.10683577},
}
Figure