An Efficient and Accurate Dynamic Sparse Training Framework Based on Parameter-Freezing

Lei Li, Haochen Yang, Jiacheng Guo, Hongkai Yu, Minghai Qin, Tianyun ZhangAAAI 2025 · DOI

Abstract

Federated Learning (FL) enables collaborative model training across decentralized devices while preserving data privacy. However, the massive size of modern Deep Neural Networks (DNNs) presents significant communication and computation challenges for edge devices. While Dynamic Sparse Training (DST) addresses these bottlenecks by training only a subset of weights, it often struggles with "mask mismatch" in non-IID data environments. This paper presents PFFDST (Parameter Freezing-based Federated Dynamic Sparse Training), a novel framework that centralizes mask management and utilizes a two-stage parameter-freezing strategy. PFFDST demonstrates up to 6% higher accuracy than existing methods and reduces communication overhead by 4–8x for equivalent accuracy levels.

Introduction

The deployment of Federated Learning is hindered by two primary constraints: limited upload bandwidth for syncing millions of parameters and the lack of high-performance hardware on edge devices. Dynamic Sparse Training (DST) has emerged as a promising solution, maintaining a sparse model throughout the training process. However, in heterogeneous (non-IID) settings, local clients often diverge on which weights are critical, leading to unstable global models. PFFDST addresses this by shifting the responsibility of architectural evolution to the server and optimizing the training phases for maximum efficiency.

Methodology: PFFDST Framework

The PFFDST framework introduces two core innovations to stabilize and accelerate sparse federated training.

Server-side Mask Readjustment (SMR)

Unlike traditional DST where clients independently determine weight importance, PFFDST implements Server-side Mask Readjustment. The server aggregates client updates and performs pruning on the global model based on weight magnitude. It then "grows" new connections to explore the parameter space. This centralized approach acts as a filter, ensuring the sparse structure reflects global trends rather than local data biases, thereby mitigating the mask mismatch problem.

Parameter-Freezing (PF)

To further enhance efficiency, PFFDST employs a two-stage Parameter-Freezing strategy:

  1. Base Training: A sparse model is trained until the weights reach a "mature" state.
  2. Freezing and Expansion: These mature weights are frozen, and the server grows additional weights to reach the target sparsity.

During the second stage, clients only compute gradients and transmit updates for the newly grown weights. This significantly reduces the communication payload and local computation (FLOPs).

Overview of the PFFDST training stages, including mask readjustment and parameter-freezing.

Experimental Evaluation

The framework was evaluated using standard image classification benchmarks:

  • Datasets: CIFAR-10 and CIFAR-100, partitioned using a Dirichlet distribution ($\alpha = 0.1$) to simulate extreme non-IID conditions.
  • Architectures: Experiments were conducted on LeNet and ResNet-18.

Key Results and Insights

PFFDST consistently outperforms state-of-the-art sparse training methods like FedDST across multiple metrics:

  • Accuracy: On ResNet-18 with CIFAR-100, PFFDST achieved a 6% accuracy improvement over FedDST under identical resource constraints.
  • Efficiency: The framework achieved a 4–8x reduction in communication costs to reach target accuracy levels.
  • Resource Optimization: The parameter-freezing stage reduced training FLOPs by approximately 33% and halved the communication cost per round compared to standard sparse training.

The success of PFFDST highlights that while data must remain private and decentralized, architectural decisions are more robust when handled centrally. Furthermore, freezing mature weights prevents "catastrophic forgetting" in noisy federated environments, allowing for more stable convergence.

Limitations and Future Work

While effective, PFFDST introduces new hyperparameters, such as the timing of the freezing phase and the readjustment ratio, which may require tuning for different tasks. Future research will explore the application of this framework to Transformer-based architectures and Large Language Models (LLMs), as well as further optimizing client-side memory usage for extremely low-end hardware.

Summary updated

BibTeX
@inproceedings{li2025efficient,
  author = {Lei Li and Haochen Yang and Jiacheng Guo and Hongkai Yu and Minghai Qin and Tianyun Zhang},
  title = {{An Efficient and Accurate Dynamic Sparse Training Framework Based on Parameter-Freezing}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i17.34019},
}
Figure