ZeroFL: Efficient On-Device Training for Federated Learning with Local Sparsity

Xinchi Qiu, Javier Fernández-Marqués, Pedro Porto Buarque de Gusmão, Yan Gao, Titouan Parcollet, Nicholas Donald Lane

TL;DR

This paper introduces ZeroFL, a novel framework for federated learning (FL) that accelerates on-device training by leveraging high levels of sparsity. By applying local sparsification before communication, ZeroFL significantly reduces computational load and communication costs on resource-constrained devices, achieving competitive or improved accuracy compared to existing methods.

What Problem Does This Paper Address?

Federated Learning (FL) enables collaborative model training across decentralized devices (like smartphones) without sharing raw data, preserving user privacy. However, these devices are typically low-power and have limited compute and memory, making it challenging to train complex machine learning models efficiently on-device. While much research in FL focuses on improving aggregation strategies or reducing communication overhead, accelerating the actual on-device training process itself – which is repeated many times and consumes most of the client's energy – has been less explored. This paper tackles this gap by investigating how to introduce sparsity during the on-device training phase to make it more efficient.

Methods

The core of this work is the ZeroFL framework, which builds upon the SWAT (Sparse Weights and Activations Training) method. SWAT sparsifies weights and activations during both forward and backward passes of training. ZeroFL adapts this by introducing local sparsification strategies applied before the client sends its updated model to the central server. This is crucial because it allows clients to reduce the amount of data they transmit, thereby lowering communication costs and potentially improving aggregation quality by reducing "noise" from less important updates.

ZeroFL proposes three main strategies for local sparsification:

  • Top-K-Weights: Clients send only the top-K weights (based on magnitude) after local training.
  • Diff on Top-K-Weights: Clients send the difference between their updated weights and the previous global weights, but only for the top-K most significant weight updates.
  • Top-K-Weights Diff: Clients send the difference of the top-K weights.

These methods are parameterized by a "mask ratio" (rmask), which controls how many of the top weights are actually sent, allowing a trade-off between communication savings and performance.

Datasets

The authors evaluated ZeroFL on three standard datasets:

  • CIFAR-10: An image classification dataset with 10 classes.
  • FEMNIST: An extension of the MNIST dataset, partitioned by writer, with 62 classes.
  • Speech Commands: An audio classification dataset with 10 keywords.

These datasets were used to simulate federated learning scenarios with varying degrees of data heterogeneity (IID and non-IID).

Key Insights and Ideas

  • On-device training is a major bottleneck in FL: The paper highlights that the client-side training, repeated over many rounds, is the primary consumer of energy and time, making its efficiency critical.
  • Sparsity can accelerate FL on-device: Introducing high levels of sparsity (up to 95%) during local training, as proposed by ZeroFL, can significantly speed up computations.
  • FL introduces unique challenges for sparsity: Unlike centralized training where sparsity patterns tend to stabilize, in FL, client data heterogeneity can lead to more dynamic sparsity patterns, making direct application of existing methods difficult.
  • Local sparsification is key for communication efficiency: By sparsifying updates before sending them to the server, ZeroFL drastically reduces the amount of data transmitted, which is a major bottleneck in FL.
  • The location of important weights is surprisingly consistent: The analysis shows that even with diverse client data, the most important weights (top-K) tend to overlap significantly across clients, suggesting that a global sparse model could be feasible.

Key Results

  • ZeroFL, with 95% sparsity, achieved up to 2.3% higher accuracy compared to baselines on non-IID Speech Commands data.
  • For CIFAR-10 at 95% sparsity, ZeroFL achieved 1.5% higher accuracy and reduced communication costs by up to 7.4x compared to vanilla SWAT.
  • The framework demonstrated improved performance and faster convergence, especially in non-IID settings.
  • The study found that a mask ratio of around 0.1 to 0.2 often provided a good balance between performance and communication savings.

Limitations and Future Work

  • Accuracy degradation at high sparsity: While ZeroFL mitigates accuracy loss compared to direct adaptation of SWAT, there is still a noticeable drop in accuracy with very high sparsity levels (e.g., 95%) in FL, especially in non-IID settings. The paper suggests this is due to the "dilution of important information" during aggregation.
  • Hardware support: The paper notes that efficient acceleration of sparse operations relies on specific hardware support, which might not be universally available on all edge devices.
  • Future work: The authors call for further investigations into device-oriented optimizations for federated learning to enable more realistic deployments of such sparse training methodologies. They also suggest exploring the impact of different sparsification strategies and their interplay with various FL aggregation algorithms.

Summary updated

Abstract

When the available hardware cannot meet the memory and compute requirements to efficiently train high performing machine learning models, a compromise in either the training quality or the model complexity is needed. In Federated Learning (FL), nodes are orders of magnitude more constrained than traditional server-grade hardware and are often battery powered, severely limiting the sophistication of models that can be trained under this paradigm. While most research has focused on designing better aggregation strategies to improve convergence rates and in alleviating the communication costs of FL, fewer efforts have been devoted to accelerating on-device training. Such stage, which repeats hundreds of times (i.e. every round) and can involve thousands of devices, accounts for the majority of the time required to train federated models and, the totality of the energy consumption at the client side. In this work, we present the first study on the unique aspects that arise when introducing sparsity at training time in FL workloads. We then propose ZeroFL, a framework that relies on highly sparse operations to accelerate on-device training. Models trained with ZeroFL and 95% sparsity achieve up to 2.3% higher accuracy compared to competitive baselines obtained from adapting a state-of-the-art sparse training framework to the FL setting.
BibTeX
@inproceedings{qiu2022zerofl,
  author = {Xinchi Qiu and Javier Fernández-Marqués and Pedro Porto Buarque de Gusmão and Yan Gao and Titouan Parcollet and Nicholas Donald Lane},
  title = {{ZeroFL: Efficient On-Device Training for Federated Learning with Local Sparsity}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=U2ZU_SA6xbr},
}
Figure