ZeroFL: Efficient On-Device Training for Federated Learning with Local Sparsity

Xinchi Qiu, Javier Fernández-Marqués, Pedro P. B. Gusmao, Yan Gao, Titouan Parcollet, Nicholas Donald Lane

Training deep learning models on smartphones and IoT devices is a bit like trying to run a marathon while breathing through a straw. In Federated Learning (FL), we want devices to learn from local data without sharing it, but these devices are severely constrained by battery life, compute power, and memory. While much research focuses on reducing the data sent between the device and the server, the actual act of training on the device remains a massive bottleneck.

In their ICLR 2022 paper, researchers from Cambridge, Oxford, and Avignon University introduce ZeroFL, a framework designed to accelerate on-device training by embracing extreme sparsity.

TL;DR

ZeroFL is the first framework to leverage high levels of sparsity (up to 95%) to accelerate the local training phase of Federated Learning. By intelligently pruning weights and activations during training and sparsifying updates sent to the server, it achieves up to 2.3% higher accuracy than adapted state-of-the-art methods while slashing communication costs by up to 7.4x.

What Problem Does This Paper Address?

The "triplet of doom" for on-device training consists of limited compute, memory, and power. Standard Federated Learning requires devices to perform multiple training epochs locally. This is energy-intensive and slow.

While "sparse training" (training only a fraction of the model's weights) works well in centralized data centers, it struggles in FL. In a data center, the model sees the same data repeatedly, allowing the "sparse mask" (the map of which weights are active) to stabilize. In FL, clients are often "stateless"—a device might participate in one round and then disappear for a while—and their data is "non-IID" (highly varied and unique to the user). Naively applying sparse training to FL usually results in a massive 10% drop in accuracy because the important information gets "diluted" during the aggregation process at the central server.

Methods

ZeroFL builds upon a centralized technique called SWAT (Sparse Weights and Activations Training). SWAT uses a "top-K" magnitude operator: during every forward and backward pass, it only uses the weights and activations with the largest values, effectively treating the rest as zero.

The authors adapt this to the FL setting with a novel contribution: Local Sparsification of Uplink Communication. Instead of sending a dense update back to the server, ZeroFL uses three strategies to keep the communication sparse:

  1. Top-K-Weights: Clients only send the weights that remained "active" during local training.
  2. Diff on Top-K-Weights: Clients send the difference between the starting weights and the final top-K weights.
  3. Top-K-Weights Diff: Clients calculate all weight differences first, then send only the largest ones.

To prevent the model from becoming too rigid, they introduce a mask ratio ($r_{mask}$). This allows a small percentage of additional weights to be sent, providing "room to breathe" for the model to evolve across rounds.

Datasets

The researchers evaluated ZeroFL on three diverse benchmarks:

  • CIFAR-10: A standard image classification task.
  • FEMNIST: A federated version of the Extended MNIST dataset (handwritten digits/characters), partitioned by the original writers to simulate real-world data heterogeneity.
  • SpeechCommands: An audio dataset for keyword spotting (e.g., "Yes", "No", "Up", "Down"), representing a common edge AI use case.

Key Insights and Ideas

Sparsity ratio distribution across model layers over training rounds.
  • The Overlap Phenomenon: One of the most surprising findings was that even though different clients have very different data, the "important" weights (the top-K by magnitude) overlap significantly across the entire fleet of devices.
  • Stability in Chaos: Despite the "stateless" nature of FL clients, the locations of non-zero weights in the global model remain remarkably constant throughout the training rounds. As seen in the layer-wise sparsity distributions, the network naturally finds and maintains a stable sparsity structure.
  • Information Dilution: The authors identified that the primary reason sparse FL fails is that when a sparse update from one client is averaged with dense or differently-masked updates from others, the "signal" of the important weights is drowned out by "noise." ZeroFL fixes this by forcing a common sparsity structure during the uplink.

Key Results

Validation accuracy of adapted SWAT across different sparsity levels on CIFAR-10.
  • Accuracy Gains: Even at extreme sparsity levels, the framework maintains competitive performance. At 95% sparsity, ZeroFL outperformed competitive baselines by +2.3% on SpeechCommands and +1.5% on CIFAR-10.
  • Communication Efficiency: By using the Compressed Sparse Row (CSR) format for updates, ZeroFL reduced the size of the data sent to the server by 3.0x to 7.4x compared to dense models.
  • Training Acceleration: By relying on sparse operations, the framework targets the same 3.3x training speedup seen in centralized sparse training, making it feasible to train larger models on smaller batteries.

Limitations and Future Work

The authors acknowledge that while the theory for speedup is there, actual wall-clock time improvements depend heavily on hardware support. Currently, most mobile CPUs and GPUs are optimized for dense matrix math; unstructured sparsity (like that used in ZeroFL) often requires sparsity levels above 90% to see real-world speed gains on commodity hardware.

Future work could involve exploring structured sparsity (pruning entire blocks or channels), which is easier for current hardware to accelerate, or developing custom hardware kernels specifically for the ZeroFL training loop.

Summary updated

Abstract

When the available hardware cannot meet the memory and compute requirements to efficiently train high performing machine learning models, a compromise in either the training quality or the model complexity is needed. In Federated Learning (FL), nodes are orders of magnitude more constrained than traditional server-grade hardware and are often battery powered, severely limiting the sophistication of models that can be trained under this paradigm. While most research has focused on designing better aggregation strategies to improve convergence rates and in alleviating the communication costs of FL, fewer efforts have been devoted to accelerating on-device training. Such stage, which repeats hundreds of times (i.e. every round) and can involve thousands of devices, accounts for the majority of the time required to train federated models and, the totality of the energy consumption at the client side. In this work, we present the first study on the unique aspects that arise when introducing sparsity at training time in FL workloads. We then propose ZeroFL, a framework that relies on highly sparse operations to accelerate on-device training. Models trained with ZeroFL and 95% sparsity achieve up to 2.3% higher accuracy compared to competitive baselines obtained from adapting a state-of-the-art sparse training framework to the FL setting.
BibTeX
@inproceedings{qiu2022zerofl,
  author = {Xinchi Qiu and Javier Fernández-Marqués and Pedro P. B. Gusmao and Yan Gao and Titouan Parcollet and Nicholas Donald Lane},
  title = {{ZeroFL: Efficient On-Device Training for Federated Learning with Local Sparsity}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=U2ZU_SA6xbr},
}
Figure