Abstract
Adapting Foundation Models (FMs) for downstream tasks through Federated Learning (FL) emerges as a promising strategy for protecting data privacy and valuable FMs. Existing methods fine-tune FMs by allocating sub-FMs to clients in FL; however, this often leads to suboptimal performance due to insufficient tuning and inevitable error accumulations of gradients. In this paper, we propose Federated Proxy Fine-Tuning (FedPFT), a novel method enhancing FM adaptation in downstream tasks through FL via two key modules. First, the sub-FM construction module employs a layer-wise compression approach, facilitating comprehensive FM fine-tuning across all layers by emphasizing crucial neurons. Second, the sub-FM alignment module conducts two-step distillations—layer-level and neuron-level—before and during FL fine-tuning respectively, to reduce gradient error by accurately aligning the sub-FM with the FM under theoretical guarantees. Experimental results on seven commonly used datasets demonstrate the superiority of FedPFT.
Introduction
Fine-tuning Foundation Models (FMs) like BERT, RoBERTa, or ViT for specific tasks usually requires massive compute and access to sensitive data. Federated Learning (FL) solves the data privacy issue by training models locally on client devices. However, two major hurdles remain: model copyright/size and the "Layer-Drop" flaw. Existing "offsite-tuning" methods send clients a "sub-model" created by dropping intermediate layers. This leads to skipped feature transformations and "gradient error accumulation," where the sub-model's gradients drift away from the full model's needs over communication rounds.
Methodology
FedPFT introduces two primary modules to bridge the gap between the server's full FM and the clients' limited resources:
- Sub-FM Construction (Layer-wise Compression): Instead of deleting entire layers, FedPFT shrinks every layer. It focuses on the Feed-Forward Networks (FFN) within Transformer blocks, pruning the least important neurons based on "neuron saliency" ($L_2$-norm of weights). This ensures the sub-model maintains the same depth and structural skeleton as the original FM, allowing for more comprehensive fine-tuning.
- Sub-FM Alignment (Two-Step Distillation): To prevent the sub-model from drifting during training, FedPFT uses knowledge distillation in two phases. Before FL, the server aligns the sub-FM to the FM using a general public dataset. During FL, the sub-FM is periodically re-aligned. To prevent "forgetting" the specific downstream task, APoZ (Average Percentage of Zero activations) is used to identify and update only the neurons most relevant to the local data.
Experimental Results
The researchers evaluated FedPFT across seven diverse NLP and CV datasets (e.g., SST-2, CIFAR-10). Key findings include:
- Superior Performance: On BERT-base, FedPFT achieved 91.6% accuracy on SST-2, matching full-model fine-tuning (FedPETuning) and outperforming previous proxy methods like FedOT (90.4%).
- Efficiency: FedPFT reduced parameters from 81M to 47M, translating to a ~50% reduction in both computational complexity and communication overhead.
- Robustness: FedPFT maintained high performance stability in "Non-IID" scenarios, effectively handling data heterogeneity across clients through its alignment module.
Conclusion
FedPFT demonstrates that maintaining model depth through layer-wise compression and ensuring continuous alignment via distillation allows for efficient, high-performance federated fine-tuning of large FMs. Future work will explore scaling to billion-parameter models and application to generative tasks.
Summary updated