TL;DR
This paper introduces pFedES, a novel approach for personalized federated learning where clients can use different model architectures. pFedES uses a shared, lightweight "proxy" feature extractor that learns generalized knowledge, which is then combined with each client's unique, personalized model. This allows for efficient knowledge sharing and state-of-the-art accuracy with significantly reduced communication and computation costs.
What Problem Does This Paper Address?
Traditional federated learning (FL) assumes all clients use the same model architecture (model homogeneity). However, in real-world scenarios, clients often have diverse needs and capabilities:
- Data Heterogeneity: Clients' data distributions can be vastly different (non-IID).
- System Resource Heterogeneity: Clients (e.g., mobile devices) have varying computational power and network bandwidth.
- Model Ownership Heterogeneity: Organizations might want to use FL to fine-tune their proprietary models without revealing their internal structures.
These challenges lead to the field of Model-Heterogeneous Personalized Federated Learning (MHPFL). Existing MHPFL methods often struggle with:
- Requiring a public dataset for knowledge transfer, which isn't always available or representative.
- High computational and communication overheads.
- Potential risks of model structure leakage.
This paper aims to address these limitations by proposing a method that facilitates effective knowledge sharing between heterogeneous models while preserving privacy and efficiency.
Methods
The core innovation of pFedES lies in its generalized proxy feature extractor sharing mechanism. Here's how it works:
- Shared Proxy Feature Extractor: Before each client's unique, heterogeneous local model, a small, homogeneous "proxy" feature extractor is introduced. This proxy extractor is designed to be lightweight and learn generalizable features from the data.
- Iterative Training: Clients engage in an iterative training process:
- Phase 1: Freeze Proxy, Train Heterogeneous Model: The global proxy feature extractor (received from the server) is frozen. The client's local heterogeneous model is then trained using two inputs: the client's raw data and the "enhanced" data produced by passing the raw data through the frozen proxy extractor. This phase transfers global knowledge to the local model.
- Phase 2: Freeze Heterogeneous Model, Train Proxy Extractor: The client's heterogeneous local model is frozen. The client then trains its local proxy feature extractor using its raw data and the frozen heterogeneous model to predict labels. This phase allows the proxy extractor to learn personalized knowledge from the client's data.
- Proxy Extractor Aggregation: After local training, only the updated local proxy feature extractors are sent to the server. The server aggregates these proxy extractors to create an updated global proxy feature extractor. The heterogeneous local models remain on the clients.
This approach ensures that only the small proxy extractors are transmitted, significantly reducing communication costs and protecting the privacy of the clients' complex local models.
Datasets
The paper evaluates pFedES on three standard image classification datasets:
- MNIST: A dataset of handwritten digits.
- CIFAR-10: A dataset of 10 classes of small images.
- CIFAR-100: A dataset of 100 classes of small images.
For experiments, these datasets were made non-IID by assigning clients data from only a subset of classes (e.g., 2 out of 10 for CIFAR-10, 10 out of 100 for CIFAR-100). No new datasets were introduced.
Key Insights and Ideas
- The Power of a Small, Shared Component: The key insight is that a small, generalized feature extractor can act as an effective bridge for knowledge transfer between vastly different local models. It doesn't need to be complex to capture essential global patterns.
- Bidirectional Knowledge Flow: The iterative training process is crucial. It allows global knowledge to flow into the local models (by training the heterogeneous model with the frozen proxy) and local insights to inform the global knowledge base (by training the proxy with the frozen heterogeneous model).
- Efficiency Through Abstraction: By only sharing the proxy extractors, pFedES achieves remarkable reductions in communication and computation. This is a significant practical advantage for real-world FL deployments.
- Privacy by Design: The heterogeneous local models never leave the client, and only the small proxy extractors are shared. This inherently protects the proprietary structures of individual client models.
Key Results
- State-of-the-Art Accuracy: pFedES consistently achieved higher average test accuracy compared to 9 baselines across various configurations (number of clients and data distribution). It achieved up to 1.29% higher accuracy.
- Massive Cost Savings: pFedES demonstrated significant efficiency gains:
- Communication Cost Reduction: Up to 99.6% lower communication costs compared to the best baseline (FedGH).
- Computational Cost Reduction: Up to 82.9% lower computational costs compared to the best baseline (FedGH).
- Improved Individual Accuracy Distribution: pFedES showed smaller accuracy variances across clients, indicating more consistent performance.
- Theoretical Guarantees: The paper provides theoretical analysis proving that pFedES converges with a non-convex convergence rate of $O(1/T)$.
Limitations and Future Work
- Limitations:
- The paper acknowledges that while pFedES incurs low extra computational overhead, it still adds a small additional model to train on each client.
- The effectiveness of the proxy extractor's structure might be sensitive to the specific task and dataset, though the paper suggests flexibility in its design.
- Future Work:
- The paper suggests exploring different proxy feature extractor structures and their impact on performance.
- Further investigation into the theoretical properties and practical implications of the iterative training process could be beneficial.
- Extending pFedES to other machine learning tasks beyond supervised image classification is a natural next step.
Summary updated