Device-Wise Federated Network Pruning

Shangqian Gao, Junyi Li, Zeyu Zhang, Yanfu Zhang, Weidong Cai, Heng HuangCVPR 2024 · DOI

Abstract

Federated Learning (FL) faces significant challenges when deploying large-scale neural networks on resource-constrained edge devices, particularly under non-IID (non-identically and independently distributed) data conditions. This paper introduces Device-Wise Federated Network Pruning (DWNP), a novel framework that personalizes network pruning for individual devices. By utilizing a Hypernetwork and device-specific embeddings, DWNP generates tailored sub-networks that adapt to local data distributions and hardware constraints. Our approach employs Federated Bilevel Optimization (FedBiO) to simultaneously optimize model weights and pruning strategies. Experimental results on CIFAR and TinyImageNet benchmarks demonstrate that DWNP not only reduces computational costs but also frequently improves model accuracy over unpruned baselines, especially in highly heterogeneous environments.

Introduction

The proliferation of deep learning on edge devices, such as smartphones and IoT sensors, is limited by their restricted memory and computational capacity. While network pruning—specifically channel pruning—is a standard technique to mitigate these constraints, its integration into Federated Learning (FL) is non-trivial. Two primary hurdles exist: data privacy, which prevents the central server from inspecting local data to identify redundant parameters, and data heterogeneity (non-IID), where a single global pruned model often fails to perform optimally across diverse local datasets.

Traditional FL pruning methods typically aim for a "one-size-fits-all" global model. However, a model optimized for a specific geographic or demographic subset may perform poorly for others. DWNP addresses this by treating pruning as a personalized architectural search problem, allowing each device to maintain a unique sub-network architecture while still benefiting from collaborative learning.

Methods

The core of the DWNP framework is the transformation of static pruning into a dynamic, learnable process facilitated by a Hypernetwork (HN).

Hypernetwork and Device Embeddings

Instead of learning a fixed binary mask for the entire network, DWNP assigns a unique low-dimensional embedding to each participating device. The Hypernetwork takes these embeddings as input and generates a personalized pruning mask (architecture vector) for each device. This allows the system to scale to millions of devices by storing only one small Hypernetwork and a set of embeddings rather than individual models.

The DWNP framework architecture.

Two-Tier Pruning Strategy

DWNP implements a hierarchical pruning approach:

  1. Server-side sub-network: A base model is pruned to remove channels that are redundant across the entire federation, reducing global communication overhead.
  2. Device-side sub-networks: These are further refined versions of the server-side model, personalized to fit local data distributions and specific device resource budgets.

Optimization and Differentiability

To train this system, the authors propose Federated Bilevel Optimization (FedBiO). The lower-level objective optimizes the shared network weights, while the upper-level objective optimizes the Hypernetwork and embeddings. To handle the discrete nature of pruning (deciding whether to keep or remove a channel), the framework utilizes the Straight-Through Gumbel-Sigmoid trick, making the architecture search differentiable and trainable via standard backpropagation. Additionally, a FLOPs regularization term is included in the loss function to ensure the generated sub-networks adhere to predefined computational budgets.

Experimental Setup

The framework was evaluated using ResNet-18 and VGG-16 architectures on three benchmarks: CIFAR-10, CIFAR-100, and TinyImageNet. To simulate real-world data heterogeneity, the authors used a Dirichlet distribution with parameter $\alpha$. A smaller $\alpha$ (e.g., 0.05) indicates higher non-IID variance between devices.

Key Results

DWNP consistently outperformed state-of-the-art FL pruning baselines. On the CIFAR-100 dataset using ResNet-18, DWNP achieved a +1.74% accuracy improvement over the original unpruned model while reducing FLOPs by 50%. This suggests that personalized pruning can act as a regularizer, helping models specialize without overfitting to local noise.

Convergence and performance analysis of DWNP.

As shown in the performance analysis, the regularization loss converges rapidly across different Hypernetwork configurations. Furthermore, the method demonstrates significant robustness to extreme pruning; even when 70% of FLOPs are removed, the test accuracy remains comparable to the full-sized model. The communication cost analysis indicates that while the Hypernetwork adds some overhead, it remains manageable and scales efficiently with the embedding dimensions.

Discussion and Insights

A key insight from this work is the "Meta-Pruner" capability of the Hypernetwork. By observing diverse data across all devices, the HN learns the underlying relationship between data distributions and optimal architectures. Interestingly, the gains from DWNP actually increased as the number of devices grew from 10 to 50, indicating strong scalability.

However, some limitations remain. The authors observed that early layers of convolutional neural networks are less affected by device-side pruning, likely because they capture general features common to all tasks. Future research could explore more sophisticated constraints to balance pruning depth more effectively. Additionally, while communication is optimized, the server-side storage of embeddings for millions of clients could eventually present a bottleneck.

Conclusion

Device-Wise Federated Network Pruning represents a significant step toward efficient and personalized edge AI. By moving away from static global models and embracing device-specific architectures via Hypernetworks, DWNP provides a scalable solution to the dual challenges of resource constraints and data heterogeneity in Federated Learning.

Summary updated

BibTeX
@inproceedings{gao2024device,
  author = {Shangqian Gao and Junyi Li and Zeyu Zhang and Yanfu Zhang and Weidong Cai and Heng Huang},
  title = {{Device-Wise Federated Network Pruning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.01173},
}
Figure