Complement Sparsification: Low-Overhead Model Pruning for Federated Learning

Xiaopeng Jiang, Cristian BorceaAAAI 2023 · DOI

Abstract

Federated Learning (FL) allows collaborative model training without sharing raw data, but the massive size of modern deep learning models creates prohibitive communication and computational overhead for edge devices. We propose Complement Sparsification (CS), a novel pruning framework where the server and clients exchange complementary sparse updates. Unlike traditional methods that require expensive fine-tuning, CS utilizes standard training rounds to implicitly recover accuracy. Experimental results on non-IID benchmarks demonstrate that CS reduces bidirectional communication by up to 93% and client-side FLOPs by nearly 50%, while remaining within 3% of dense model accuracy and significantly outperforming existing state-of-the-art baselines.

Introduction

Modern deep learning models are notoriously over-parameterized, leading to two primary bottlenecks in Federated Learning: massive communication costs for model synchronization and heavy computational loads for resource-constrained IoT and mobile devices. While model pruning—removing unimportant weights—is a common solution, applying it to FL is challenging. Existing techniques often require a data-dependent fine-tuning phase that the server cannot perform, or they only reduce one-way communication (client-to-server), leaving the download phase heavy. Furthermore, aggressive pruning in non-IID (non-Independent and Identically Distributed) settings often leads to substantial accuracy degradation.

The Complement Sparsification Framework

The core innovation of CS is the collaborative relationship between the server and its clients. Instead of clients training a subset of the global model, they focus on the weights the server has discarded.

The CS workflow: The server prunes the global model, and clients train the removed weights (the complement) to capture local trends.

Server and Client Roles

  1. Server-Side Pruning: After aggregating updates into a dense model, the server prunes weights with the lowest magnitudes to create a Global Sparse Model. It then transmits this sparse model along with a binary pruning mask to the clients.
  2. Client-Side Complementary Training: Clients do not simply train the received sparse model. Instead, they train the complement: the specific weights that were zeroed out by the server. This allows clients to capture local data trends that might have been overlooked by the global pruning criteria.
  3. Aggregation and Growth: In subsequent rounds, the server integrates these complementary updates back into the global model using an aggregation ratio ($\eta'$) greater than 1. This "growth" mechanism allows previously pruned weights to regain importance, ensuring the model's sparse structure can dynamically evolve to find the optimal sub-network.

Experimental Evaluation

The researchers evaluated CS using the LEAF framework on two non-IID benchmarks: Twitter (Sentiment Analysis) and FEMNIST (Image Classification).

Key Results

CS demonstrates superior efficiency across multiple metrics:

  • Communication Efficiency: Achieved a 93.2% reduction in client-to-server data transfer and up to an 80% reduction in server-to-client transfer.
  • Computational Savings: Reduced client-side training FLOPs by 29.1% to 49.3%.
  • Competitive Accuracy: Maintained performance within a minimal gap of dense models, reaching 74.3% accuracy on Sentiment Analysis (vs. 76.9% for dense FL) and 72.5% on Image Classification (vs. 76.3% for dense FL).
Accuracy performance over 500 rounds on FEMNIST, comparing CS against vanilla FL and existing pruning methods like PruneFL and PQSU.

As shown in the figure above, CS significantly outperforms state-of-the-art baselines like PruneFL and PQSU. On the complex FEMNIST dataset, these baselines experienced accuracy drops of 15-17% compared to CS, highlighting the robustness of the complementary training approach in heterogeneous data environments.

Discussion and Insights

A key finding is that CS often produces smoother learning curves than vanilla FL. In non-IID settings, dense models can be sensitive to noisy local updates. By pruning low-magnitude weights every round, the server effectively filters out this noise, leading to more stable global convergence. Additionally, the dynamic nature of the pruning mask allows the model to explore different architectures throughout training, rather than being locked into a static sparse structure.

Conclusion and Future Work

Complement Sparsification provides a high-efficiency path for Federated Learning on edge devices. By leveraging the synergy between global pruning and local complementary training, it eliminates the need for explicit fine-tuning while slashing overhead. Future research will focus on optimizing hyperparameter tuning for the aggregation ratio and developing specialized hardware kernels to further translate theoretical FLOP reductions into real-world wall-clock time speedups on mobile GPUs.

Summary updated

BibTeX
@inproceedings{jiang2023complement,
  author = {Xiaopeng Jiang and Cristian Borcea},
  title = {{Complement Sparsification: Low-Overhead Model Pruning for Federated Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i7.25977},
}
Figure