FedPFT: Federated Proxy Fine-Tuning of Foundation Models

Zhaopeng Peng, Xiaoliang Fan, Yufan Chen, Zheng Wang, Shirui Pan, Chenglu Wen, Ruisheng Zhang, Cheng Wang

Introduction

Foundation Models (FMs), such as BERT, GPT, and ViT, have revolutionized machine learning by providing powerful, pre-trained representations that can be adapted to a wide range of downstream tasks. However, fine-tuning these models in privacy-sensitive or resource-constrained environments remains a significant challenge. Federated Learning (FL) offers a decentralized training paradigm, but applying it to FMs introduces two critical hurdles: proprietary concerns that prevent model owners from sharing full weights, and the massive memory and compute requirements that exceed the capabilities of most edge devices.

Existing approaches, such as "offsite-tuning," attempt to mitigate these issues by sending a smaller "sub-model" to clients. These sub-models are typically created by dropping entire layers. Unfortunately, this creates a structural mismatch; intermediate layers of the original FM are never updated, and the sub-model's gradients eventually drift away from the ideal optimization path of the full model, leading to substantial performance loss.

The FedPFT Framework

To address these limitations, the authors introduce Federated Proxy Fine-Tuning (FedPFT). This framework allows distributed clients to adapt massive FMs without accessing the full model or compromising private data. FedPFT relies on two core innovations: layer-wise compression and a two-step distillation process.

Layer-wise Compression (Sub-FM Construction)

Unlike previous methods that skip entire "chapters" of a model by dropping layers, FedPFT performs a "summary" of every layer. It focuses on the Feed-Forward Networks (FFN) within transformer blocks, using the L2-norm of weights to measure "neuron saliency." By pruning less important neurons across every layer rather than removing layers entirely, the resulting proxy model maintains a consistent structural correspondence with the original FM. This ensures that gradient flow remains intact throughout the entire depth of the architecture.

Two-Step Distillation (Sub-FM Alignment)

To prevent the proxy model from drifting during the federated process, FedPFT employs a dual-phase alignment strategy:

  1. Pre-FL Alignment: Before the federated training begins, the server utilizes a public dataset to distill knowledge from the full FM into the sub-FM. This alignment occurs at both the output and weight levels, providing a high-quality starting point for the clients.
  2. During-FL Alignment: To maintain synchronization, the sub-FM is periodically re-aligned with the full FM during the training process. To prevent "catastrophic forgetting" of the specific downstream tasks learned by clients, the authors use Average Percentage of Zero activations (APoZ). This metric identifies and updates only the neurons that are less active for the specific task, preserving the specialized knowledge contributed by the decentralized participants.

Theoretical and Empirical Validation

The researchers provide theoretical proofs (Theorems 1 and 2) demonstrating that as long as the proxy model remains within a specific error bound of the full model, the full model is guaranteed to converge toward an optimal solution.

Empirically, FedPFT was evaluated across seven datasets in Natural Language Processing (NLP) and Computer Vision (CV), including SST-2, QNLI, MNLI, CIFAR-100, and Oxford Flowers. The results highlight three major advantages:

  • High Accuracy: FedPFT consistently outperformed leading baselines like FedOT. On the QNLI dataset using BERT, FedPFT achieved 87.5% accuracy, significantly higher than FedOT’s 83.9% and remarkably close to the 88.4% achieved by full-model fine-tuning.
  • Resource Efficiency: The framework reduced the parameter count for clients from 81M to 47M. This reduction effectively halves the computational and communication costs, making FM adaptation viable for edge devices.
  • Robustness: In Non-IID scenarios—where data distribution varies significantly across clients—FedPFT maintained a higher performance floor than existing methods, proving its utility for real-world, heterogeneous data environments.

Conclusion and Future Directions

FedPFT represents a significant advancement in making Foundation Models accessible and trainable within federated ecosystems. By prioritizing layer-wise structural integrity and selective alignment, it bridges the gap between efficiency and performance. While current evaluations focus on base-scale models and classification tasks, future work will explore scaling the framework to "Large" or "Huge" variants (such as LLaMA) and extending the methodology to generative tasks like dialogue and summarization.

Summary updated

BibTeX
@inproceedings{peng2024fedpft,
  author = {Zhaopeng Peng and Xiaoliang Fan and Yufan Chen and Zheng Wang and Shirui Pan and Chenglu Wen and Ruisheng Zhang and Cheng Wang},
  title = {{FedPFT: Federated Proxy Fine-Tuning of Foundation Models}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=yKON4cTfd1},
}
Figure