Do We Need to Directly Access the Source Datasets for Domain Generalization?

Junkun Yuan, Xu Ma, Defang Chen, Kun Kuang, Fei Wu, Lanfen Lin

Abstract

Domain generalization (DG) aims to learn from multiple known source domains a model that can generalize well to unknown target domains. The existing DG methods usually exploit the fusion of shared multi-source data to train a generalizable model. However, tremendous data is distributed across lots of places nowadays that can not be shared due to privacy policies. In this paper, we tackle the problem of federated domain generalization where the source datasets can only be accessed and learned locally for privacy protection. We propose a novel framework called Collaborative Semantic Aggregation and Calibration (CSAC) to enable this challenging problem. To fully absorb multi-source semantic information while avoiding unsafe data fusion, we conduct data-free semantic aggregation by fusing the models trained on the separated domains layer-by-layer. To address the semantic dislocation problem caused by domain shift, we further design cross-layer semantic calibration with an attention mechanism to align each semantic level and enhance domain invariance. We unify multi-source semantic learning and alignment in a collaborative way by repeating the semantic aggregation and calibration alternately, keeping each dataset localized, and the data privacy is carefully protected. Extensive experiments show the significant performance of our method in addressing this challenging problem.

Introduction

In deep learning, we often face a "distribution shift": a model trained on photos of dogs might fail when it encounters a sketch of a dog. Domain Generalization (DG) aims to solve this by training on multiple "source" domains (e.g., photos, cartoons, and paintings) to learn a universal representation of a "dog."

Comparison of standard Domain Generalization (shared source domains) and Federated Domain Generalization (separated source domains).

The catch? In the real world, data is often locked behind privacy walls (e.g., medical records in different hospitals). We can't simply pool all the data into one server. Federated Learning (FL) keeps data local, but standard FL algorithms like FedAvg struggle with DG because they assume the data is somewhat similar across clients. When domains are highly diverse, simple model averaging fails to capture the "invariance" needed to handle a completely new, unknown target domain.

Methodology: The CSAC Framework

The authors propose Collaborative Semantic Aggregation and Calibration (CSAC), which moves beyond simple parameter averaging through three main steps:

The CSAC framework architecture, featuring local semantic acquisition, data-free semantic aggregation, and cross-layer semantic calibration.
  1. Local Semantic Acquisition: Each local site trains its own model. They use "label smoothing" to prevent the models from becoming over-confident in their specific local biases.
  2. Data-Free Semantic Aggregation: Instead of a simple average, the server fuses models layer-by-layer. Crucially, they use semantic divergence-based weights. If a local model’s parameters are very different from the average, it’s given more weight. The logic: these "outlier" models contain unique distribution information that the global model needs to "absorb" to become truly generalizable.
  3. Cross-Layer Semantic Calibration: This is the "secret sauce." Due to domain shift, the same semantic concept (like a "nose") might be learned in layer 3 of one model but layer 4 of another—a problem called semantic dislocation. CSAC uses a dual-attention mechanism (position-wise and channel-wise) to align features across different layers between the global fused model and local models, ensuring the model learns consistent concepts regardless of the domain.

Experimental Setup

The researchers evaluated CSAC on four major benchmarks:

  • PACS: The gold standard for DG (Photos, Art, Cartoons, Sketches).
  • VLCS: Real-world images from four different datasets (Pascal, LabelMe, Caltech, Sun).
  • Rotated MNIST: Handwritten digits rotated at various angles to simulate domain shift.
  • Office-Caltech-Home: A complex, 7-domain dataset constructed by the authors by merging common classes from Office-Caltech and Office-Home.

Key Results and Insights

  • Superior Performance: On the PACS dataset, CSAC achieved an average accuracy of 83.81%, outperforming the standard FedAvg (82.36%) and the previous state-of-the-art federated method FedDG (82.15%).
  • Generalization Leap: In a specific test on the "Sketch" domain (using Art, Cartoon, and Photo as sources), the fused CSAC model improved accuracy by 13% to 50% compared to the individual local models before fusion.
  • Efficiency: CSAC is significantly faster than its competitors. It runs in roughly 4.6 hours on PACS, compared to 11.8 hours for FedDG, primarily because it avoids the complex "distribution bank" building process.

Theoretical Insights

  • Outliers are Assets: In traditional federated learning, we often try to minimize the influence of divergent clients. CSAC treats divergent models as rich sources of unique domain information, giving them higher weights during fusion to ensure the global model doesn't just become a "lowest common denominator."
  • The "Where" Matters as Much as the "What": By acknowledging that different domains might store information in different layers (semantic dislocation), the authors show that cross-layer alignment is essential for robust generalization.

Conclusion

The authors note that CSAC’s power is most evident when using larger, more complex networks (like ResNet-18). On very small networks, the performance gain was less dramatic, suggesting that the "semantic calibration" needs a certain amount of architectural "headroom" to work its magic. Future directions include scaling this to scenarios with thousands of separated source datasets and exploring its application in highly sensitive fields like finance and medical diagnostics.

Summary updated

Abstract

Domain generalization (DG) aims to learn from multiple known source domains a model that can generalize well to unknown target domains. The existing DG methods usually exploit the fusion of shared multi-source data to train a generalizable model. However, tremendous data is distributed across lots of places nowadays that can not be shared due to privacy policies. In this paper, we tackle the problem of federated domain generalization where the source datasets can only be accessed and learned locally for privacy protection. We propose a novel framework called Collaborative Semantic Aggregation and Calibration (CSAC) to enable this challenging problem. To fully absorb multi-source semantic information while avoiding unsafe data fusion, we conduct data-free semantic aggregation by fusing the models trained on the separated domains layer-by-layer. To address the semantic dislocation problem caused by domain shift, we further design cross-layer semantic calibration with an attention mechanism to align each semantic level and enhance domain invariance. We unify multi-source semantic learning and alignment in a collaborative way by repeating the semantic aggregation and calibration alternately, keeping each dataset localized, and the data privacy is carefully protected. Extensive experiments show the significant performance of our method in addressing this challenging problem.
BibTeX
@article{yuan2023do,
  author = {Junkun Yuan and Xu Ma and Defang Chen and Kun Kuang and Fei Wu and Lanfen Lin},
  title = {{Do We Need to Directly Access the Source Datasets for Domain Generalization?}},
  journal = {IEEE Trans. Knowl. Data Eng.},
  year = {2023},
  doi = {10.1109/tkde.2023.3271851},
}
Figure