Abstract
Federated learning (FL) is an emerging machine learning paradigm that allows multiple parties to train a shared model collaboratively in a privacy-preserving manner. Existing horizontal FL methods generally assume that the FL server and clients hold the same model structure. However, due to system heterogeneity and the need for personalization, enabling clients to hold models with diverse structures has become an important direction. Existing model-heterogeneous FL approaches often require publicly available datasets and incur high communication and/or computational costs, which limit their performances. To address these limitations, we propose a simple but effective Federated Global prediction Header (FedGH) approach. It is a communication and computation-efficient model-heterogeneous FL framework which trains a shared generalized global prediction header with representations extracted by heterogeneous extractors for clients' models at the FL server. The trained generalized global prediction header learns from different clients. The acquired global knowledge is then transferred to clients to substitute each client's local prediction header. We derive the non-convex convergence rate of FedGH. Extensive experiments on two real-world datasets demonstrate that FedGH achieves significantly more advantageous performance in both model-homogeneous and -heterogeneous FL scenarios compared to seven state-of-the-art personalized FL models, beating the best-performing baseline by up to 8.87% (for model-homogeneous FL) and 1.83% (for model-heterogeneous FL) in terms of average test accuracy, while saving up to 85.53% of communication overhead.
Introduction
Standard Federated Learning (FL) algorithms, such as FedAvg, typically operate under the assumption of model homogeneity, where all participating clients must utilize an identical model architecture. However, in practical deployments, clients often possess varying computational capabilities—ranging from high-performance servers to resource-constrained IoT devices. This "system heterogeneity" necessitates a more flexible approach where clients can employ different model structures suited to their local hardware.
Current solutions for model-heterogeneous FL often rely on knowledge distillation using auxiliary public datasets or impose significant communication and computational burdens. Furthermore, the challenge of Non-IID (not identically and independently distributed) data persists, where local data distributions vary significantly across clients, hindering the convergence and generalization of a global model.
The FedGH Framework
To overcome these challenges, we introduce FedGH (Federated Global Header). The core innovation lies in decoupling the model into two distinct components: a heterogeneous Feature Extractor and a homogeneous Prediction Header.
As illustrated in the figure above, the FedGH workflow follows a four-step process:
- Local Training: Each client trains its local model (extractor + header) on its private dataset.
- Representation Extraction: Clients compute a Local Averaged Representation (LAR) for each class. This vector represents the average latent embedding for all samples of a specific class (e.g., the "average panda" representation).
- Global Header Training: The server aggregates these LARs to train a Generalized Global Header. This header learns to map representations from various client extractors to their respective global labels.
- Header Replacement: The optimized global header is sent back to clients to replace their local headers, effectively injecting global knowledge into the local models.
This architecture ensures that while feature extractors remain private and tailored to local hardware, the global header acts as a bridge for knowledge sharing without the need for raw data or full model weight transfers.
Experimental Evaluation
We evaluated FedGH against seven state-of-the-art baselines, including FedProto, LG-FedAvg, and FedKD. Experiments were conducted on two benchmark datasets:
- CIFAR-10: Configured in a Non-IID setting where each client holds only 2 out of 10 classes.
- CIFAR-100: A more challenging setting where each client holds 10 out of 100 classes.
The evaluation covered both model-homogeneous scenarios and model-heterogeneous scenarios (using five different CNN architectures across clients).
Key Results and Discussion
FedGH demonstrates superior performance in both accuracy and efficiency. In model-homogeneous tests on CIFAR-100, FedGH outperformed the best baseline by 8.87%. In heterogeneous settings, it maintained a significant lead of 1.83%.
As shown in the convergence plots, FedGH (dashed line with stars) consistently achieves higher test accuracy and faster convergence compared to other methods. Beyond accuracy, FedGH is remarkably efficient; on CIFAR-100, it reduced communication overhead by 85.53% compared to FedProto. This efficiency stems from only transmitting the small prediction header and class-averaged representations rather than full model parameters.
Conclusion
FedGH provides a robust and efficient solution for federated learning in heterogeneous environments. By focusing on a generalized global header and abstract representations, it achieves state-of-the-art accuracy while drastically reducing communication costs. Future work will explore more sophisticated representation methods beyond simple averaging to mitigate information loss and investigate hybrid header fusion techniques to better balance global generalization with local personalization.
Summary updated