TL;DR
FedGH (Federated Global Header) is a streamlined framework that allows Federated Learning (FL) participants to use entirely different model architectures. By sharing only class-averaged data representations and a small "prediction header" instead of full model weights, FedGH achieves superior accuracy and slashes communication costs by up to 85% compared to state-of-the-art methods—all without requiring any auxiliary public datasets.
What Problem Does This Paper Address?
Standard Federated Learning algorithms like FedAvg assume a "one size fits all" approach: every participant (client) must use the exact same model architecture. In reality, this is rarely feasible. A powerful server and a low-power IoT sensor have vastly different computational resources. This is known as system heterogeneity.
Existing attempts to solve this usually fall into two traps:
- Public Data Dependency: They require a shared, unlabeled public dataset to perform "knowledge distillation," which isn't always available or representative of private data.
- High Overhead: They require clients to send massive amounts of data or perform complex computations that drain battery and bandwidth.
Furthermore, data across clients is often Non-IID (not identically and independently distributed), meaning one client might only see "cats" while another only sees "dogs," making it hard for local models to generalize.
Methods
The authors propose FedGH, which relies on a simple but clever architectural split:
- The Feature Extractor: This is the "body" of the model that turns raw data into abstract mathematical representations. In FedGH, this part is heterogeneous—each client can choose an extractor (like a small CNN or a large ResNet) that fits their device's power.
- The Prediction Header: This is the "head" of the model that makes the final classification. This part is homogeneous across all clients.
The FedGH Workflow: 1. Local Training: Clients train their full local models on their private data. 2. Representation Extraction: Instead of sending model weights, clients compute a Local Averaged Representation (LAR)—a single average vector for each class they own (e.g., the "average cat" representation). 3. Global Header Training: The server receives these LARs and uses them to train a Generalized Global Header. This header learns the global relationship between representations and labels from all clients. 4. Header Replacement: The server sends the updated global header back to the clients, who use it to replace their local headers.
This "model splicing" allows the global knowledge to be injected directly into local models without ever sharing the private feature extractors.
Datasets
The researchers evaluated FedGH using two staple computer vision benchmarks, modified to simulate difficult real-world conditions:
- CIFAR-10: Divided into a Non-IID setting where each client only holds 2 out of the 10 possible classes.
- CIFAR-100: A more complex task where each client holds only 10 out of 100 classes.
They tested these across both model-homogeneous (everyone uses the same architecture) and model-heterogeneous (clients use five different sizes of CNNs) scenarios.
Key Insights and Ideas
- Privacy through Abstraction: By sending LARs (averages of latent embeddings) rather than raw data or full gradients, FedGH adds a layer of privacy. It is significantly harder to "reverse-engineer" original images from a class-wide average vector than from specific model weights.
- Communication Efficiency: A prediction header is much smaller than a full model. Sending only the header and a few LAR vectors drastically reduces the bits flying across the network.
- Global Knowledge, Local Body: The "aha" moment is that the server doesn't need to see the feature extractor to improve it. By providing a better "head" (the global header), the server forces the local "body" (the extractor) to produce representations that align with a global understanding of the data.
- Hyperparameter Robustness: Experiments showed that FedGH is remarkably insensitive to the learning rate of the global header, making it easier to deploy in the wild without tedious tuning.
Key Results
FedGH consistently outperformed seven state-of-the-art baselines (including FedProto, LG-FedAvg, and FedKD):
- Accuracy Boost: In model-homogeneous tests on CIFAR-100, FedGH beat the best baseline by 8.87%. In heterogeneous settings, it maintained a lead of 1.83%.
- Efficiency: On the CIFAR-100 dataset, FedGH reduced communication overhead by 85.53% compared to FedProto while reaching higher accuracy.
- Convergence Speed: FedGH reached target accuracy levels in significantly fewer rounds than its competitors, proving it is not just accurate, but fast.
Limitations and Future Work
The authors identify two main areas for improvement:
- Information Distortion: Using a single average vector (LAR) to represent an entire class might lose important nuances, especially if the local data for that class is very diverse. They suggest exploring more "integrated" representations.
- Personalization vs. Generalization: Currently, the global header completely replaces the local one. The authors suggest that "fusing" the global and local headers might provide a better balance between global knowledge and local personalization.
Summary updated