Accelerating Wireless Federated Learning via Nesterov's Momentum and Distributed Principle Component Analysis

Yanjie Dong, Luya Wang, Yuanfang Chi, Jia Wang, Haijun Zhang, Fei Richard Yu, Victor C. M. Leung, Xiping HuarXiv 2023 · DOI · arXiv

Introduction

The rise of "Edge Intelligence" has positioned Federated Learning (FL) as a cornerstone for training global models on decentralized data. However, deploying FL in wireless environments presents significant challenges: the communication bottleneck caused by high-dimensional gradient updates and the inherent "chaos" of wireless channels, such as the near-far effect and non-i.i.d. data distributions. This paper introduces PCA-AWFL, a framework designed to overcome these hurdles by integrating one-shot distributed Principal Component Analysis (PCA) with Nesterov’s accelerated gradient descent.

System Architecture and Methods

The PCA-AWFL framework addresses communication efficiency and robustness through three primary mechanisms: a realistic wireless system model, dimension reduction, and momentum-based optimization.

1. Wireless Federated Learning Setup

The system consists of a central server and $N$ workers. Unlike traditional FL models that assume ideal conditions, this framework accounts for non-uniform worker locations and varying channel qualities.

The PCA-AWFL system architecture showing the exchange of model parameters (w_k) and gradients between a central server and multiple workers.

As illustrated above, the server broadcasts model weights $w_k$, and workers return local gradients. To mitigate the "near-far effect," where workers at different distances experience varying signal strengths, the authors employ truncated channel inversion. This ensures the server receives an unbiased global gradient estimate despite fading and noise, preventing the system from being limited by the "weakest link."

2. One-shot Distributed PCA

To slash communication costs, the researchers introduce a one-shot distributed PCA step before the training phase begins. This identifies the most significant features of the local datasets, allowing workers to project high-dimensional data $d_0$ into a lower-dimensional space $\hat{d}_0$.

This transformation, represented as $\hat{A}_n$, directly reduces the size of every gradient uploaded during the subsequent training rounds. This "one-shot" approach is highly efficient as it only requires a single round of coordination before the iterative training starts.

3. Nesterov’s Momentum

The "A" in PCA-AWFL stands for "Accelerated." By incorporating Nesterov’s momentum, the algorithm uses historical gradient information to refine the search direction. This effectively tracks the "flow of intelligence" across the non-convex loss landscapes typical of deep learning, allowing the model to converge much faster than standard gradient descent.

Experimental Evaluation

The authors validated PCA-AWFL using three distinct datasets: FMNIST (fashion images), CIFAR-10 (object categories), and AWE (a real-world industrial dataset from high-speed railway maintenance).

Key Results

  • Communication Efficiency: On the FMNIST dataset, reducing dimensions from 784 to 500 achieved a 36.22% reduction in communication expenditure per frame, with a negligible (0.34%) drop in testing accuracy.
  • Faster Convergence: In numerical tests, PCA-AWFL reached stability in 20,000 to 30,000 frames, whereas standard PCA-WFL and other benchmarks often required 50,000 frames or more to reach the same level of performance.
  • Industrial Robustness: The algorithm demonstrated superior accuracy on the noisy AWE industrial dataset, proving its effectiveness in practical, non-ideal wireless environments where channel noise and fading are prevalent.

Conclusion

PCA-AWFL offers a powerful solution for wireless edge intelligence. By combining dimension reduction with accelerated optimization, it significantly reduces communication overhead while speeding up convergence. While the accelerated version may converge to a slightly larger error floor, the gains in speed and efficiency make it a compelling choice for real-time industrial and mobile applications. Future work will explore dynamic dimension adjustment and defenses against potentially malicious workers in the distributed PCA process.

Summary updated

Abstract

A wireless federated learning system is investigated by allowing a server and workers to exchange uncoded information via orthogonal wireless channels. Since the workers frequently upload local gradients to the server via bandwidth-limited channels, the uplink transmission from the workers to the server becomes a communication bottleneck. Therefore, a one-shot distributed principle component analysis (PCA) is leveraged to reduce the dimension of uploaded gradients such that the communication bottleneck is relieved. A PCA-based wireless federated learning (PCA-WFL) algorithm and its accelerated version (i.e., PCA-AWFL) are proposed based on the low-dimensional gradients and the Nesterov's momentum. For the non-convex loss functions, a finite-time analysis is performed to quantify the impacts of system hyper-parameters on the convergence of the PCA-WFL and PCA-AWFL algorithms. The PCA-AWFL algorithm is theoretically certified to converge faster than the PCA-WFL algorithm. Besides, the convergence rates of PCA-WFL and PCA-AWFL algorithms quantitatively reveal the linear speedup with respect to the number of workers over the vanilla gradient descent algorithm. Numerical results are used to demonstrate the improved convergence rates of the proposed PCA-WFL and PCA-AWFL algorithms over the benchmarks.
BibTeX
@misc{dong2023accelerating,
  author = {Yanjie Dong and Luya Wang and Yuanfang Chi and Jia Wang and Haijun Zhang and Fei Richard Yu and Victor C. M. Leung and Xiping Hu},
  title = {{Accelerating Wireless Federated Learning via Nesterov's Momentum and Distributed Principle Component Analysis}},
  year = {2023},
  eprint = {2303.17885},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2303.17885},
}
Figure