Abstract
Fine-tuning large-scale Foundation Models (FMs) in Federated Learning (FL) environments is often prohibited by the significant computational and communication requirements of resource-constrained clients. This paper introduces FedPFT (Federated Proxy Fine-Tuning), a framework that utilizes a "thin but deep" proxy model to facilitate efficient fine-tuning. By employing layer-wise compression instead of layer-dropping and a two-step knowledge distillation process for alignment, FedPFT mitigates gradient mismatch and information loss. Experimental results across NLP and CV tasks demonstrate that FedPFT reduces overhead by approximately 50% while maintaining performance within 1% of full-model fine-tuning.
Introduction
Foundation Models (FMs) like BERT and ViT have set new benchmarks in machine learning, but their massive parameter counts pose a challenge for Federated Learning. Standard FL requires clients to download, update, and upload model parameters, which often exceeds the memory and bandwidth of local devices.
Existing solutions usually attempt "layer-dropping," where middle layers are removed to create a smaller model. However, this approach suffers from two primary flaws: Information Loss, as discarding intermediate layers prevents learning complex features, and Gradient Mismatch, where the structural differences cause client updates to diverge from the full model's ideal path.
Methodology
FedPFT addresses these challenges through two core modules: Sub-FM Construction and Sub-FM Alignment.
Sub-FM Construction
Instead of removing entire layers, FedPFT maintains the full depth of the Transformer architecture. It employs layer-wise compression, making each layer "thinner" by pruning less important neurons in the Feed-Forward Networks (FFN) based on their saliency. This ensures the proxy model (sub-FM) retains the same depth as the original FM.
Sub-FM Alignment
To resolve the gradient mismatch problem, FedPFT uses a two-step Knowledge Distillation (KD) strategy:
- Pre-FL Alignment: The server aligns the sub-FM with the full FM using a general dataset before training begins.
- In-FL Neuron-Level Distillation: During training, clients identify active neurons using the Average Percentage of Zero activations (APoZ) metric. Only the least active neurons are updated during alignment to protect the specialized knowledge gained from local private data.
Key Results and Insights
FedPFT was evaluated on seven datasets across NLP (e.g., SST-2, MNLI) and CV (e.g., CIFAR-10, Flowers).
- Efficiency: The framework achieves a ~50% reduction in both computational complexity and communication costs compared to full-model fine-tuning.
- Performance: On BERT and RoBERTa, FedPFT consistently outperformed baselines like FedOT. It achieved accuracy within 0.5%–1% of the full 81M parameter model, despite the proxy using only 47M parameters.
- Robustness: The method proved highly effective in Non-IID scenarios, where data distributions vary significantly across clients.
Conclusion
FedPFT provides a scalable solution for fine-tuning massive models on edge devices. By prioritizing model depth and implementing a robust alignment strategy, it bridges the gap between efficiency and performance in federated environments. Future work will explore applying FedPFT to even larger Large Language Models (LLMs) and more complex generative tasks.
Summary updated