Introduction
Federated Learning (FL) has emerged as a powerful paradigm for training machine learning models across distributed clients without compromising data privacy. However, a significant challenge remains: Federated Domain Generalization (FedDG). In real-world scenarios, data across clients is often non-i.i.d. (not independently and identically distributed), representing different domains (e.g., photos vs. sketches). The goal of FedDG is to train a global model that can generalize effectively to completely unseen domains at test time.
Existing FedDG methods typically focus on aligning feature distributions or final predictions (logits). However, these approaches often fail to capture the fine-grained "reasoning process" of the model—specifically, how the relationship between the feature extractor and the classifier evolves across different domains. This paper introduces FedDIM, a novel framework that utilizes a Decision Insight Matrix (DIM) to align these internal interactions, fostering domain-invariant learning.
Methodology
The core of the FedDIM approach is the Decision Insight Matrix, which captures the element-wise interactions between feature vectors and classifier weights.
The Decision Insight Matrix (DIM)
In a standard classification layer, the logit for a class is the dot product of the feature vector and the class's weight vector. FedDIM decomposes this operation. Instead of looking at the scalar sum, it examines the individual element-wise products. This matrix, $I$, represents how much each specific feature dimension contributes to the classification decision for each class. By aligning these matrices, the model ensures that the "reasoning" behind a classification remains consistent, regardless of the input domain.
Federated Alignment
- Local Computation: Each client $k$ computes a local mean insight matrix for each class based on its private data.
- Global Aggregation: The server aggregates these local matrices to form a global "anchor" insight matrix. A momentum-based update is employed to ensure stability against data heterogeneity.
- Consistency Regularization: During local training, a regularization loss ($L_{im}$) is added to the standard cross-entropy loss ($L_{ce}$). This term penalizes deviations between an individual sample's insight matrix and the global class-wise anchor, forcing the model to focus on domain-invariant semantic cues.
Experimental Setup
FedDIM was evaluated on four prominent domain generalization benchmarks:
- RotatedMNIST: Digits rotated from 0° to 75°.
- PACS: Images spanning Photo, Art, Cartoon, and Sketch domains.
- VLCS: Data from four distinct image datasets (Caltech101, LabelMe, SUN09, VOC2007).
- OfficeHome: 65 object classes across four diverse artistic styles.
The experiments covered both simple (one-domain-one-client) and complex (one-domain-multi-client) federated settings to test robustness and scalability.
Results and Analysis
FedDIM demonstrates superior performance across all benchmarks. In the one-domain-one-client scenario, it achieved an average accuracy of 82.66%, significantly outperforming the standard FedAvg baseline (80.56%) and other state-of-the-art methods like FedIIR.
Generalization and Loss Surfaces
A key finding is that FedDIM helps the model converge to "flatter" minima in the loss landscape. In deep learning theory, flatter regions are strongly correlated with better generalization to unseen data. Visualization of the loss surfaces confirms that FedDIM produces a much broader, more stable convergence region compared to the sharper, more brittle minima of FedAvg.
Representation Learning
t-SNE visualizations further support the effectiveness of the method. FedDIM produces tighter, more distinct class clusters on unseen domains. This indicates that the model has successfully learned to ignore domain-specific noise (like the difference between a photo and a sketch) in favor of robust semantic features.
Conclusion
FedDIM addresses the challenges of Federated Domain Generalization by aligning the internal decision-making process of the model via the Decision Insight Matrix. This approach is computationally efficient, privacy-preserving, and leads to superior generalization on unseen data. Future work includes extending this framework to scenarios with heterogeneous label spaces and exploring its application to Vision Transformer architectures.
Summary updated