Bayesian Federated Learning over Wireless Networks

Seunghoon Lee, Chanho Park, Song-Nam Hong, Yonina C. Eldar, Namyoon LeearXiv 2020 · arXiv

Abstract

Federated Learning (FL) faces significant challenges in wireless environments due to communication bandwidth constraints, channel noise, and data heterogeneity. While 1-bit quantization methods like signSGD reduce bandwidth, they are often sensitive to noise and non-IID data. This paper introduces Bayesian Federated Learning (BFL) and its practical implementation, Scalable-BFL (SBFL). By treating gradient aggregation as a Minimum Mean-Squared Error (MMSE) estimation problem, SBFL incorporates local gradient statistics and channel state information into a non-linear aggregation function. Experimental results on MNIST and synthetic datasets demonstrate that SBFL achieves up to a 5x speedup in convergence and superior accuracy compared to traditional signSGD in noisy wireless settings.

1. Introduction

In the world of Federated Learning (FL), communication is the ultimate bottleneck. To mitigate this, algorithms like signSGD have popularized 1-bit quantization to slash bandwidth requirements. However, these methods often stumble in real-world deployments where wireless channels are noisy and data across devices is wildly inconsistent (Non-IID). Standard FL assumes that a central server can perfectly receive updates, but in reality, devices transmit over heterogeneous wireless links—some with strong signals and others plagued by fading and noise.

Lee et al. propose a more sophisticated way to aggregate these "noisy votes." By treating gradient aggregation as a Bayesian estimation problem rather than a simple majority vote, the authors demonstrate significant gains in training speed and model accuracy.

2. Problem Formulation

Traditional signSGD uses 1-bit quantization (sending only the sign of the gradient) and aggregates them via majority voting. This approach is suboptimal for two primary reasons:

  1. Channel Heterogeneity: It treats a "vote" from a device with a terrible, noisy connection the same as a "vote" from a device with a perfect signal.
  2. Statistical Ignorance: It ignores prior knowledge about the distribution of local gradients, which varies significantly in Non-IID settings where a device training on one set of labels produces a different statistical "fingerprint" than another.

3. Proposed Method: Bayesian Federated Learning

The core contribution of this work is the shift from simple voting to Minimum Mean-Squared Error (MMSE) estimation. The authors propose two frameworks:

3.1 Bayesian Federated Learning (BFL)

The "ideal" BFL algorithm assumes the server knows the joint distribution of all local gradients and the exact channel conditions. It calculates the optimal estimate of the global gradient by combining the 1-bit quantized inputs with the likelihood of channel noise.

3.2 Scalable-BFL (SBFL)

Since knowing the full joint distribution is computationally impossible for large models, the authors propose SBFL, a practical variant characterized by:

  • Simplified Priors: Each device models its gradient elements as independent and identically distributed (iid) Gaussian or Laplacian variables.
  • Minimal Overhead: Instead of sending a full covariance matrix, devices send their 1-bit quantized gradient plus just two scalars: the mean ($\mu$) and variance ($\nu^2$) of their local gradients.
  • Non-linear Aggregation: The server uses a closed-form, non-linear aggregation function based on the tanh (hyperbolic tangent) activation. This implies that the optimal aggregator can be implemented as a shallow, two-layer neural network at the edge server.

4. Experimental Evaluation

The authors evaluated their approach using synthetic datasets for linear regression and the MNIST handwritten digit dataset. To simulate a realistic FL environment, they used a Non-IID setup where each user only held data from two specific digit classes.

4.1 Key Results

  • Faster Convergence: In synthetic tests, SBFL reached target loss levels in significantly fewer rounds. For MNIST, SBFL with a Gaussian prior reached 90% test accuracy in 139 rounds, whereas signSGD required 638 rounds—a nearly 5x speedup.
  • Higher Accuracy: In heterogeneous wireless settings, SBFL consistently outperformed signSGD. On MNIST, SBFL achieved a test accuracy of ~95%, while signSGD plateaued around 90% or lower depending on the learning rate.
  • Robustness to Noise: SBFL proved much more resilient to "faded" signals. Because the aggregator accounts for the channel coefficient ($h_k$), it naturally de-prioritizes updates from devices currently experiencing deep fades.

5. Discussion and Insights

The research highlights that the server should not just "average" updates; it should "estimate" the true gradient. By knowing a device's local gradient variance and its current channel Signal-to-Noise Ratio (SNR), the server can weigh that device's contribution intelligently. It is notable how much performance is recovered by sending just two extra numbers ($\mu$ and $\nu^2$). This side information allows the server to denoise the 1-bit signals effectively without significantly increasing communication overhead.

Furthermore, the derivation shows that the optimal Bayesian aggregator for Gaussian priors naturally takes the form of a tanh function, providing a theoretical link between Bayesian estimation and neural network architectures.

6. Limitations and Future Work

SBFL assumes gradients follow Gaussian or Laplacian distributions. While often true due to the Central Limit Theorem, this might not hold for all architectures. Additionally, the algorithm requires accurate Channel State Information (CSI) at the server. Future directions include exploring over-the-air computation to aggregate gradients analogly and applying this Bayesian framework to federated multi-task learning.

Summary updated

Abstract

Federated learning is a privacy-preserving and distributed training method using heterogeneous data sets stored at local devices. Federated learning over wireless networks requires aggregating locally computed gradients at a server where the mobile devices send statistically distinct gradient information over heterogenous communication links. This paper proposes a Bayesian federated learning (BFL) algorithm to aggregate the heterogeneous quantized gradient information optimally in the sense of minimizing the mean-squared error (MSE). The idea of BFL is to aggregate the one-bit quantized local gradients at the server by jointly exploiting i) the prior distributions of the local gradients, ii) the gradient quantizer function, and iii) channel distributions. Implementing BFL requires high communication and computational costs as the number of mobile devices increases. To address this challenge, we also present an efficient modified BFL algorithm called scalable-BFL (SBFL). In SBFL, we assume a simplified distribution on the local gradient. Each mobile device sends its one-bit quantized local gradient together with two scalar parameters representing this distribution. The server then aggregates the noisy and faded quantized gradients to minimize the MSE. We provide a convergence analysis of SBFL for a class of non-convex loss functions. Our analysis elucidates how the parameters of communication channels and the gradient priors affect convergence. From simulations, we demonstrate that SBFL considerably outperforms the conventional sign stochastic gradient descent algorithm when training and testing neural networks using MNIST data sets over heterogeneous wireless networks.
BibTeX
@misc{lee2020bayesian,
  author = {Seunghoon Lee and Chanho Park and Song-Nam Hong and Yonina C. Eldar and Namyoon Lee},
  title = {{Bayesian Federated Learning over Wireless Networks}},
  year = {2020},
  eprint = {2012.15486},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2012.15486},
}
Figure