Abstract
Limited computation and communication capabilities of clients pose significant challenges in federated learning (FL) over resource-limited edge nodes. A potential solution to this problem is to deploy off-the-shelf sparse learning algorithms that train a binary sparse mask on each client with the expectation of training a consistent sparse server mask. However, as we investigate in this paper, such naive deployments result in a significant accuracy drop compared to FL with dense models, especially under low client's resource budget. In particular, our investigations reveal a serious lack of consensus among the trained masks on clients, which prevents convergence on the server mask and potentially leads to a substantial drop in model performance. Based on such key observations, we propose federated lottery aware sparsity hunting (FLASH), a unified sparse learning framework to make the server win a lottery in terms of a sparse sub-model, which can greatly improve performance under highly resource-limited client settings. Moreover, to address the issue of device heterogeneity, we leverage our findings to propose hetero-FLASH, where clients can have different target sparsity budgets based on their device resource limits. Extensive experimental evaluations with multiple models on various datasets (both IID and non-IID) show superiority of our models in yielding up to $\mathord{\sim}10.1\%$ improved accuracy with $\mathord{\sim}10.26\times$ fewer communication costs, compared to existing alternatives, at similar hyperparameter settings.
Introduction
Federated Learning (FL) is the primary strategy for training machine learning models on edge devices without compromising user privacy. However, edge devices—such as smartwatches or IoT sensors—are notoriously resource-constrained. While sparse learning (training only a small fraction of a model's weights) appears to be an ideal solution for reducing memory and bandwidth requirements, naive implementations often result in a significant accuracy decline.
This research identifies the root cause of this failure as a lack of sparsity consensus. In centralized training, sparse learning successfully identifies a "winning lottery ticket"—a small sub-network that performs as well as the full model. In a federated setting, however, different clients often disagree on which weights are important. This disagreement, particularly severe in deeper neural network layers, prevents the global model from converging.
The FLASH Framework
We propose FLASH (Federated Lottery Aware Sparsity Hunting), a two-stage framework designed to achieve consensus without sacrificing the benefits of sparsity.
1. Mask Initialization and Sensitivity Awareness
Instead of random initialization, FLASH utilizes a data-driven warm-up phase. The server coordinates a brief training period with a subset of clients to calculate layer sensitivity. This process identifies which layers require more parameters to function effectively, resulting in a global initial sparse mask that aligns client training from the start.
2. Training Strategies: SPDST and JMWST
FLASH offers two distinct paths for sparse training:
- SPDST (Static Pre-Defined Sparsity Training): The global mask is frozen after initialization. Clients only train the weights allowed by this mask, offering the highest communication efficiency.
- JMWST (Joint Mask Weight Sparse Training): The mask is allowed to evolve. To ensure alignment, the server performs "sub-sampling" at the end of each round to prune the global model back to the target density.
3. Hetero-FLASH
To address real-world device heterogeneity, Hetero-FLASH allows clients to train at different sparsity levels based on their specific hardware limits while still contributing to a unified global sparse structure.
Experimental Results
The framework was rigorously tested on standard benchmarks (MNIST, CIFAR-10, CIFAR-100, TinyImageNet) using ResNet18 and VGG16 models.
Accuracy at Low Density
FLASH significantly outperforms existing methods like ZeroFL at extreme sparsity levels. As illustrated in Figure 1, FLASH maintains high test accuracy even when 95% or more of the weights are removed, whereas naive methods often collapse.
Communication Efficiency
By enforcing consensus, FLASH achieves superior performance with a fraction of the communication cost. Compared to ZeroFL, FLASH provides up to ~10.1% improved accuracy while requiring ~10.26× less communication.
Conclusion
FLASH represents a significant advancement in making federated learning viable for resource-constrained edge devices. By prioritizing sparsity consensus through sensitivity-aware initialization and joint optimization, FLASH bridges the performance gap between sparse and dense models. Future research will focus on formal theoretical grounding and extending the framework to large-scale foundation models.
Summary updated