Abstract
Federated Learning (FL) faces significant challenges with domain shifts in heterogeneous data, degrading performance. Traditional domain generalization aims to learn domain-invariant features, but the federated nature of model averaging often limits this due to its linear aggregation of local learning. To address this, we propose a robust framework, coined as hypernetwork-based Federated Fusion (hFedF), using hypernetworks for non-linear aggregation, facilitating generalization to unseen domains. Our method employs client-specific embeddings and gradient alignment techniques to manage domain generalization effectively. Evaluated in both zero-shot and few-shot settings, hFedF demonstrates superior performance in handling domain shifts. Comprehensive comparisons on PACS, Office-Home, and VLCS datasets show that hFedF consistently achieves the highest in-domain and out-of-domain accuracy with reliable predictions. Our study contributes significantly to the under-explored field of Federated Domain Generalization (FDG), setting a new benchmark for performance in this area.
Introduction
In Federated Learning (FL), the objective is to train a robust machine learning model across decentralized devices without centralizing raw data. However, domain shift remains a critical obstacle. For instance, if a model is trained on MRI data from one hospital, it may fail to generalize to a different hospital where imaging protocols or equipment differ.
A recent study from researchers at ETH Zurich and KAIST AI, titled "Hypernetwork-Driven Model Fusion for Federated Domain Generalization," addresses this by introducing hFedF. By replacing traditional linear averaging with a hypernetwork-based non-linear aggregation, the framework enables models to generalize more effectively across diverse and unseen data sources.
The Challenge of Federated Domain Generalization (FDG)
Standard FL algorithms, such as FedAvg, typically utilize a weighted linear average of model weights from various clients. While computationally efficient, this approach is often "blind" to the nuances of different data domains. In the context of Federated Domain Generalization (FDG), the goal is to train on several "source" domains (e.g., photos and cartoons) so that the model performs well on a completely "unseen" target domain (e.g., sketches).
Linear averaging tends to "wash out" domain-specific features essential for building a robust representation. Privacy constraints further complicate this, as clients cannot share data to align distributions directly. The central challenge is effectively fusing knowledge from diverse clients without direct data access or oversimplifying the aggregation process.
Methodology: Hypernetwork-based Federated Fusion (hFedF)
The core innovation of hFedF is the use of a central Hypernetwork to manage model fusion. Instead of a single global model, the server maintains a hypernetwork that generates tailored parameters for each client based on unique embeddings.
Key Components:
- Non-linear Aggregation: The hypernetwork outputs weights for the client’s local network (e.g., a CNN). This allows for a flexible, non-linear fusion of knowledge that captures the richness of local representations better than simple averaging.
- Gradient Alignment (GradAlign): To stabilize training in a federated setup, the authors introduced a technique that calculates the cosine similarity between a client’s update and the average update direction. Updates that deviate significantly from the consensus are down-weighted to mitigate "client drift."
- Exponential Moving Average (EMA): EMA is applied to the hypernetwork weights across communication rounds to ensure smooth convergence and parameter stability.
- Dynamic Embeddings: The system learns unique client identifiers (embeddings) during training, allowing the hypernetwork to adapt its weight generation to specific data characteristics.
Experimental Results
The researchers evaluated hFedF on standard benchmarks: PACS (Art, Cartoon, Photo, Sketch), Office-Home, and VLCS. Using a "leave-one-domain-out" strategy, the model was trained on three domains and tested on the fourth, unseen domain.
Key Findings:
- Superior Accuracy: On the PACS dataset, hFedF achieved an out-of-domain (OOD) accuracy of 54.6%, outperforming FedAvg (51.3%) and specialized FDG methods like FedSR (48.9%).
- Few-Shot Adaptation: In scenarios where a few samples from the target domain were available, hFedF demonstrated superior adaptability, suggesting it learns more flexible features.
- Reliability: hFedF showed better "uncertainty awareness." When making incorrect predictions, it did so with lower confidence than standard methods, making it more reliable for real-world deployment.
- Quasi-Global Models: Despite generating client-specific weights, hFedF resulted in less weight divergence than local training, creating a "quasi-global" model with shared underlying logic.
Conclusion and Future Work
While hFedF sets a new benchmark for FDG, it introduces higher computational and memory overhead on the central server due to the complexity of hypernetworks. Future research may explore "partial" hypernetworks that only generate weights for specific layers to improve efficiency. Additionally, integrating hFedF with large-scale pre-trained foundation models remains a promising direction for scaling these benefits to even more complex tasks.
Summary updated