FedPFT: Federated Proxy Fine-Tuning of Foundation Models

Zhaopeng Peng, Xiaoliang Fan, Yufan Chen, Zheng Wang, Shirui Pan, Chenglu Wen, Ruisheng Zhang, Cheng Wang

Abstract

Adapting Foundation Models (FMs) for downstream tasks through Federated Learning (FL) emerges as a promising strategy for protecting data privacy and valuable FMs. Existing methods fine-tune FMs by allocating sub-FMs to clients in FL; however, this often leads to suboptimal performance due to insufficient tuning and inevitable error accumulations of gradients. In this paper, we propose Federated Proxy Fine-Tuning (FedPFT), a novel method enhancing FM adaptation in downstream tasks through FL via two key modules. First, the sub-FM construction module employs a layer-wise compression approach, facilitating comprehensive FM fine-tuning across all layers by emphasizing crucial neurons. Second, the sub-FM alignment module conducts two-step distillations—layer-level and neuron-level—before and during FL fine-tuning respectively, to reduce gradient error by accurately aligning the sub-FM with the FM under theoretical guarantees. Experimental results on seven commonly used datasets demonstrate the superiority of FedPFT.

Introduction

Fine-tuning Foundation Models (FMs) like BERT, RoBERTa, or ViT for specific tasks usually requires massive compute and access to sensitive data. Federated Learning (FL) solves the data privacy issue by training models locally on client devices. However, two major hurdles remain: model copyright/size and the "Layer-Drop" flaw. Existing "offsite-tuning" methods send clients a "sub-model" created by dropping intermediate layers. This leads to skipped feature transformations and "gradient error accumulation," where the sub-model's gradients drift away from the full model's needs over communication rounds.

Comparison of layer-drop vs. layer-wise compression and the resulting model disparity over communication rounds.

Methodology

FedPFT introduces two primary modules to bridge the gap between the server's full FM and the clients' limited resources:

  1. Sub-FM Construction (Layer-wise Compression): Instead of deleting entire layers, FedPFT shrinks every layer. It focuses on the Feed-Forward Networks (FFN) within Transformer blocks, pruning the least important neurons based on "neuron saliency" ($L_2$-norm of weights). This ensures the sub-model maintains the same depth and structural skeleton as the original FM, allowing for more comprehensive fine-tuning.
  2. Sub-FM Alignment (Two-Step Distillation): To prevent the sub-model from drifting during training, FedPFT uses knowledge distillation in two phases. Before FL, the server aligns the sub-FM to the FM using a general public dataset. During FL, the sub-FM is periodically re-aligned. To prevent "forgetting" the specific downstream task, APoZ (Average Percentage of Zero activations) is used to identify and update only the neurons most relevant to the local data.
The FedPFT framework architecture, illustrating the server-client interaction and the two key modules: Sub-FM Construction and Sub-FM Alignment.

Experimental Results

The researchers evaluated FedPFT across seven diverse NLP and CV datasets (e.g., SST-2, CIFAR-10). Key findings include:

  • Superior Performance: On BERT-base, FedPFT achieved 91.6% accuracy on SST-2, matching full-model fine-tuning (FedPETuning) and outperforming previous proxy methods like FedOT (90.4%).
  • Efficiency: FedPFT reduced parameters from 81M to 47M, translating to a ~50% reduction in both computational complexity and communication overhead.
  • Robustness: FedPFT maintained high performance stability in "Non-IID" scenarios, effectively handling data heterogeneity across clients through its alignment module.

Conclusion

FedPFT demonstrates that maintaining model depth through layer-wise compression and ensuring continuous alignment via distillation allows for efficient, high-performance federated fine-tuning of large FMs. Future work will explore scaling to billion-parameter models and application to generative tasks.

Summary updated

Abstract

Adapting Foundation Models (FMs) for down- stream tasks through Federated Learning (FL) emerges a promising strategy for protecting data privacy and valuable FMs. Existing methods fine- tune FM by allocating sub-FM to clients in FL, however, leading to suboptimal performance due to insufficient tuning and inevitable error accumula- tions of gradients. In this paper, we propose Feder- ated Proxy Fine-Tuning (FedPFT), a novel method enhancing FMs adaptation in downstream tasks through FL by two key modules. First, the sub-FM construction module employs a layer-wise com- pression approach, facilitating comprehensive FM fine-tuning across all layers by emphasizing those crucial neurons. Second, the sub-FM alignment module conducts a two-step distillations—layer- level and neuron-level—before and during FL fine- tuning respectively, to reduce error of gradient by accurately aligning sub-FM with FM under theo- retical guarantees. Experimental results on seven commonly used datasets (i.e., four text and three vi- sion) demonstrate the superiority of FedPFT. Our code is available at https://github.com/pzp-dzd/FedPFT.
BibTeX
@inproceedings{peng2024fedpft,
  author = {Zhaopeng Peng and Xiaoliang Fan and Yufan Chen and Zheng Wang and Shirui Pan and Chenglu Wen and Ruisheng Zhang and Cheng Wang},
  title = {{FedPFT: Federated Proxy Fine-Tuning of Foundation Models}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=yKON4cTfd1},
}
Figure