Abstract
This paper introduces AgeTop-k, a novel gradient compression strategy designed for Over-the-Air Federated Learning (OTA-FL). By integrating gradient magnitude with the "Age of Information" (AoI), the strategy ensures that model updates are both numerically significant and timely. This approach prevents specific parameters from becoming "stale," a common failure mode in standard sparsification techniques, thereby accelerating convergence and improving final model accuracy.
Introduction
In Federated Learning (FL), a central server aggregates updates from multiple clients to train a global model. To enhance scalability, Over-the-Air (OTA) computation leverages the superposition property of wireless signals, allowing simultaneous update aggregation. However, OTA-FL is constrained by the limited number of available orthogonal wireless waveforms compared to the high dimensionality of modern neural networks. This necessitates gradient sparsification, typically implemented as a "Top-k" selection of the largest gradient magnitudes.
Standard Top-k selection is inherently greedy; it consistently ignores parameters with smaller magnitudes. Over time, these neglected parameters become stale, leading to reconstruction errors that stall convergence. While randomized alternatives exist, they often lack the directional precision required for efficient training. AgeTop-k addresses this by introducing a fairness mechanism based on update frequency.
The AgeTop-k Framework
The proposed AgeTop-k strategy utilizes a two-stage selection process managed by the edge server to optimize the use of limited wireless resources.
As illustrated in the architecture above, the process involves:
- Magnitude Filtering: The server identifies a candidate set of $r$ gradient entries with the largest magnitudes from its global tracking vector.
- Age Prioritization: From this candidate set, the server selects $k$ entries ($k < r$) with the highest Age of Information (AoI).
AoI tracks the time elapsed since a specific parameter was last updated. The server maintains an age vector that increments for skipped parameters and resets to zero upon update. By broadcasting this selection (the sparsification matrix $\hat{S}^t$) to clients, the server ensures that updates are both significant and overdue for a refresh.
Experimental Evaluation
The strategy was validated using two standard benchmarks: EMNIST (handwritten character recognition) and CIFAR-10 (object classification using ResNet-18). The experiments simulated a non-i.i.d. environment with 20 clients to reflect realistic edge computing challenges.
As shown in the performance plots, AgeTop-k consistently outperforms standard Top-k, Random-k, and rTop-k. On the CIFAR-10 dataset, AgeTop-k achieved a final test accuracy of approximately 69%, significantly higher than standard Top-k (~66%) and Age-k (~62%).
Key Insights
- Mitigating Gradient Starvation: AgeTop-k acts as a fairness mechanism, ensuring every significant parameter eventually contributes to the global model, preventing the "starvation" of low-magnitude but essential coordinates.
- The Primacy of Magnitude: Results indicate that selecting purely based on age (Age-k) performs as poorly as random selection. Magnitude must remain the primary filter, with age serving as the optimal tie-breaker among significant candidates.
- Candidate Ratio Trade-off: There is a "Goldilocks zone" for the candidate set size ($r$). If $r$ is too small, the system reverts to greedy Top-k behavior. If $r$ is too large, the server may prioritize stale but irrelevant low-magnitude gradients, degrading accuracy.
- Practicality: AgeTop-k requires minimal overhead, as the server only needs to maintain a simple vector of integers (ages), making it highly suitable for resource-constrained edge environments.
Conclusion
AgeTop-k provides a robust solution for the scalability challenges of OTA-FL. By balancing numerical significance with update freshness, it achieves superior convergence rates and accuracy compared to existing sparsification methods. Future work will explore adaptive tuning of the $r$ and $k$ hyperparameters to respond to real-time wireless channel conditions.
Summary updated