TL;DR
This paper introduces CSAC, a novel framework for Federated Domain Generalization (FDG) that tackles the challenge of training robust models across diverse datasets while strictly preserving data privacy. CSAC achieves this by collaboratively aggregating and calibrating models trained locally on separated datasets, demonstrating strong generalization performance comparable to methods that require data sharing.
What Problem Does This Paper Address?
The core problem is Federated Domain Generalization (FDG). Traditional Domain Generalization (DG) aims to train models that perform well on unseen target domains by learning from multiple known source domains. However, many existing DG methods rely on fusing data or representations from these source domains. This poses a significant privacy challenge in real-world scenarios where data is distributed across various locations (e.g., hospitals, user devices) and cannot be shared due to privacy regulations.
FDG specifically addresses this by requiring models to learn from these distributed, private source datasets locally, without direct data sharing. This creates a dilemma: how to effectively learn domain-invariant features and achieve good generalization when direct data fusion is impossible, and each local dataset might have its own biases or peculiarities. The paper highlights two key challenges: 1) hindered learning of multi-source semantic information due to data separation, and 2) potential for negative generalization if models are trained on biased local datasets.
Methods
The proposed framework, Collaborative Semantic Aggregation and Calibration (CSAC), tackles FDG through a two-pronged approach:
- Local Semantic Acquisition: Each distributed source dataset is used to train a local model. This step focuses on extracting semantic information layer-by-layer from the local data distribution. Label smoothing is used to prevent over-confident models.
- Data-Free Semantic Aggregation: Instead of fusing raw data, CSAC fuses the trained models layer-by-layer. This is a "data-free" approach, meaning no actual data is transmitted. The aggregation is weighted based on the semantic divergence of each local model's parameters from the average, giving more importance to models with distinct distributions. This helps absorb multi-source semantic information without direct data fusion.
- Cross-Layer Semantic Calibration: This is the core innovation for addressing the "semantic dislocation" problem (where semantic information from different domains gets mixed across layers due to domain shift). CSAC aligns semantic features between the aggregated (fused) model and each local model.
- Attention Mechanism: An attention mechanism is employed to calculate semantic similarities between cross-layer feature pairs (from the fused model and local models). This allows the model to focus on semantically related features and down-weight unrelated ones, ensuring precise alignment.
- Auxiliary Retraining Loss: An additional loss is used to prevent catastrophic forgetting of knowledge from individual domains during calibration.
- Collaborative Iteration: The semantic aggregation and calibration processes are repeated alternately. This iterative process allows for continuous refinement of the model, progressively enhancing domain invariance and generalization.
Crucially, only model parameters are transmitted between clients and the server, ensuring data privacy.
Datasets
The paper evaluates CSAC on several standard benchmark datasets for domain generalization and adaptation tasks:
- PACS: A dataset for object recognition with 7 categories across 4 domains (Art, Cartoon, Sketch, Photo).
- VLCS: Another object recognition dataset with 5 classes from 4 domains (Pascal, LabelMe, Caltech, Sun).
- Rotated MNIST: A simulated digit dataset with 6 domains created by rotating the original MNIST images.
- Office-Caltech-Home: A constructed dataset merging common classes from Office-Caltech and Office-Home datasets, resulting in 7 domains.
The experiments use a "leave-one-domain-out" strategy, where one domain is held out as the target, and the rest are used as distributed source domains.
Key Insights and Ideas
- Model Fusion for Privacy: The core idea of fusing models rather than data is a critical enabler for privacy-preserving federated learning in DG. This allows knowledge extraction without exposing sensitive information.
- Layer-wise Semantic Alignment: The paper hypothesizes and demonstrates that deep models learn semantic information layer-by-layer. This insight is key to addressing semantic dislocation by aligning features at corresponding semantic levels across domains.
- Semantic Divergence Weighting: Dynamically weighting the aggregation of local models based on their semantic divergence from the average is crucial. It prevents dominant or outlier domains from unduly influencing the aggregated model, leading to a more balanced representation.
- Attention for Precise Alignment: The use of an attention mechanism for cross-layer semantic calibration is a sophisticated way to ensure that only truly semantically related features are aligned, preventing noise and improving robustness.
- Iterative Refinement: The alternating nature of semantic aggregation and calibration is a powerful iterative process that allows the model to progressively improve its generalization capabilities.
Key Results
- State-of-the-Art Performance: CSAC achieves state-of-the-art or highly competitive results on the tested datasets (PACS, VLCS, Rotated MNIST, Office-Caltech-Home) for federated domain generalization.
- Outperforming Shared Data Methods: On PACS, CSAC with separated source data even outperforms some DG methods that do use shared source data, highlighting the effectiveness of its approach.
- Scalability with More Domains: CSAC shows improved performance as the number of source domains increases, particularly on the Office-Caltech-Home dataset with 7 domains. This suggests that having more diverse semantic information aids the alignment process.
- Robustness: Sensitivity analysis shows CSAC is robust to the hyperparameter $\lambda$ used for semantic calibration.
- Efficiency: While FedDG is computationally inefficient due to its distribution bank building, CSAC's runtime is comparable to FedAvg, making it practical.
Limitations and Future Work
- Network Size: The paper notes that CSAC's full potential, especially the benefits of semantic calibration and attention, might require larger network architectures to fully manifest, as observed in experiments with Rotated MNIST.
- Future Outlook: The authors suggest that their work sheds light on the promising direction of collaboratively training generalizable models from potentially thousands of separated source datasets, which is crucial for many privacy-sensitive applications. They envision future work exploring this large-scale scenario.
Summary updated