Training deep learning models on smartphones and IoT devices is a bit like trying to run a marathon while breathing through a straw. In Federated Learning (FL), we want devices to learn from local data without sharing it, but these devices are severely constrained by battery life, compute power, and memory. While much research focuses on reducing the data sent between the device and the server, the actual act of training on the device remains a massive bottleneck.
In their ICLR 2022 paper, researchers from Cambridge, Oxford, and Avignon University introduce ZeroFL, a framework designed to accelerate on-device training by embracing extreme sparsity.
TL;DR
ZeroFL is the first framework to leverage high levels of sparsity (up to 95%) to accelerate the local training phase of Federated Learning. By intelligently pruning weights and activations during training and sparsifying updates sent to the server, it achieves up to 2.3% higher accuracy than adapted state-of-the-art methods while slashing communication costs by up to 7.4x.
What Problem Does This Paper Address?
The "triplet of doom" for on-device training consists of limited compute, memory, and power. Standard Federated Learning requires devices to perform multiple training epochs locally. This is energy-intensive and slow.
While "sparse training" (training only a fraction of the model's weights) works well in centralized data centers, it struggles in FL. In a data center, the model sees the same data repeatedly, allowing the "sparse mask" (the map of which weights are active) to stabilize. In FL, clients are often "stateless"—a device might participate in one round and then disappear for a while—and their data is "non-IID" (highly varied and unique to the user). Naively applying sparse training to FL usually results in a massive 10% drop in accuracy because the important information gets "diluted" during the aggregation process at the central server.
Methods
ZeroFL builds upon a centralized technique called SWAT (Sparse Weights and Activations Training). SWAT uses a "top-K" magnitude operator: during every forward and backward pass, it only uses the weights and activations with the largest values, effectively treating the rest as zero.
The authors adapt this to the FL setting with a novel contribution: Local Sparsification of Uplink Communication. Instead of sending a dense update back to the server, ZeroFL uses three strategies to keep the communication sparse:
- Top-K-Weights: Clients only send the weights that remained "active" during local training.
- Diff on Top-K-Weights: Clients send the difference between the starting weights and the final top-K weights.
- Top-K-Weights Diff: Clients calculate all weight differences first, then send only the largest ones.
To prevent the model from becoming too rigid, they introduce a mask ratio ($r_{mask}$). This allows a small percentage of additional weights to be sent, providing "room to breathe" for the model to evolve across rounds.
Datasets
The researchers evaluated ZeroFL on three diverse benchmarks:
- CIFAR-10: A standard image classification task.
- FEMNIST: A federated version of the Extended MNIST dataset (handwritten digits/characters), partitioned by the original writers to simulate real-world data heterogeneity.
- SpeechCommands: An audio dataset for keyword spotting (e.g., "Yes", "No", "Up", "Down"), representing a common edge AI use case.
Key Insights and Ideas
- The Overlap Phenomenon: One of the most surprising findings was that even though different clients have very different data, the "important" weights (the top-K by magnitude) overlap significantly across the entire fleet of devices.
- Stability in Chaos: Despite the "stateless" nature of FL clients, the locations of non-zero weights in the global model remain remarkably constant throughout the training rounds. As seen in the layer-wise sparsity distributions, the network naturally finds and maintains a stable sparsity structure.
- Information Dilution: The authors identified that the primary reason sparse FL fails is that when a sparse update from one client is averaged with dense or differently-masked updates from others, the "signal" of the important weights is drowned out by "noise." ZeroFL fixes this by forcing a common sparsity structure during the uplink.
Key Results
- Accuracy Gains: Even at extreme sparsity levels, the framework maintains competitive performance. At 95% sparsity, ZeroFL outperformed competitive baselines by +2.3% on SpeechCommands and +1.5% on CIFAR-10.
- Communication Efficiency: By using the Compressed Sparse Row (CSR) format for updates, ZeroFL reduced the size of the data sent to the server by 3.0x to 7.4x compared to dense models.
- Training Acceleration: By relying on sparse operations, the framework targets the same 3.3x training speedup seen in centralized sparse training, making it feasible to train larger models on smaller batteries.
Limitations and Future Work
The authors acknowledge that while the theory for speedup is there, actual wall-clock time improvements depend heavily on hardware support. Currently, most mobile CPUs and GPUs are optimized for dense matrix math; unstructured sparsity (like that used in ZeroFL) often requires sparsity levels above 90% to see real-world speed gains on commodity hardware.
Future work could involve exploring structured sparsity (pruning entire blocks or channels), which is easier for current hardware to accelerate, or developing custom hardware kernels specifically for the ZeroFL training loop.
Summary updated