FedSR: A Simple and Effective Domain Generalization Method for Federated Learning

A. Tuan Nguyen, Philip Torr, Ser-Nam LimNeurIPS 2022 · OpenReview

TL;DR

FedSR is a novel framework that enables Federated Learning (FL) models to generalize to entirely new, unseen users or "domains" without compromising privacy. By enforcing "simple" data representations through L2-norm and conditional mutual information regularizers, it achieves state-of-the-art performance on domain generalization tasks without ever requiring clients to share or compare their data.

What Problem Does This Paper Address?

Federated Learning (FL) allows multiple clients (like hospitals or mobile phone users) to train a shared model without ever sharing their raw data. However, standard FL has a major blind spot: it assumes the data seen during training is representative of what the model will see in the future.

In the real world, a model trained on data from ten hospitals might fail when deployed at an eleventh hospital because of "domain shift"—differences in equipment, patient demographics, or lighting. This is the realm of Domain Generalization (DG). While many DG techniques exist for centralized training, they usually require looking at data from all domains simultaneously to find "invariant" features. In a privacy-preserving FL setting, this is impossible because clients cannot see each other's data or even their intermediate representations. FedSR aims to bridge this gap: providing robust generalization while respecting the decentralized nature of FL.

Methods

The authors propose FedSR (Federated Simple Representation). The core philosophy is that "simpler" representations of data are less likely to overfit to domain-specific noise (like the background of an image) and more likely to capture the essential features needed for a task.

To achieve this, they introduce two key regularizers into the local training objective of each client:

  1. L2-norm Regularizer ($L^{L2R}$): This penalizes the magnitude of the data representation. Mathematically, the authors show this acts as an "implicit marginal alignment." By encouraging every client to center their data representations around zero, the clients' distributions become aligned with each other by proxy, without ever needing to communicate.
  2. Conditional Mutual Information (CMI) Regularizer ($L^{CMI}$): This restricts the amount of information the representation can carry about the input data, given the label. It forces the model to discard "spurious correlations" (e.g., a specific camera's color tint) and retain only the "essential information" needed to predict the label.

Crucially, these regularizers are computed locally. The central server only performs standard parameter averaging (FedAVG), making the method easy to drop into existing FL pipelines.

Datasets

The authors evaluated FedSR on four widely recognized benchmarks for domain generalization:

  • RotatedMNIST: Handwritten digits rotated at various angles (0° to 75°), where each angle represents a different domain.
  • PACS: A classic DG dataset containing four domains: Art painting, Cartoon, Photo, and Sketch.
  • OfficeHome: 15,500 images of everyday objects across four domains: Art, Clipart, Product, and Real-world.
  • DomainNet: A massive, challenging dataset with over 580,000 images across 345 classes and 6 domains (e.g., Quickdraw, Infograph).

Key Insights and Ideas

  • Simplicity as a Proxy for Invariance: The "aha moment" here is that you don't need to explicitly compare domains to find invariant features. If you force the representation to be "simple" and "label-focused" locally, you naturally gravitate toward features that are consistent across all domains.
  • Alignment via Reference: Instead of aligning Client A to Client B, FedSR aligns every client to a common "reference" distribution (a zero-centered Gaussian). This achieves the same goal of cross-domain alignment without any data leakage.
  • Theoretical Grounding: The paper provides a formal connection between these simple regularizers and the complex divergence-minimization techniques used in centralized DG, proving that FedSR is doing "implicit" representation alignment.

Key Results

FedSR consistently outperformed standard FL baselines and existing decentralized DG methods:

  • Superior Accuracy: On the massive DomainNet benchmark, FedSR achieved 41.3% average accuracy, outperforming the standard FedAVG (39.6%) and the previous state-of-the-art FL method, FedADG (40.4%). This strong performance is echoed on the RotatedMNIST dataset, where FedSR achieves a leading 96.7% average accuracy among FL methods.
Results on RotatedMNIST
  • Competitive with Centralized SOTA: Remarkably, FedSR often performed as well as—or better than—top-tier centralized DG methods. This is highly notable because centralized methods have the "unfair" advantage of seeing all data at once. For instance, on the PACS dataset, FedSR achieves an 83.7% average accuracy, edging out the centralized DIRT-GAN baseline (83.62%).
Results on PACS dataset
  • Robustness: The performance gains were consistent across different architectures (from simple CNNs to ResNet-50) and varying levels of task difficulty.

Limitations and Future Work

  • Source Domain Diversity: Like all DG methods, FedSR’s success depends on the diversity of the source clients. If the training clients are too similar, the model may still struggle with a radically different target domain.
  • Probabilistic Mapping: The CMI regularizer requires a probabilistic representation (sampling during training), which adds a small amount of computational overhead and architectural complexity compared to standard deterministic models.
  • Hyperparameter Sensitivity: While the authors used a standard tuning protocol, the balance between the two regularizers ($\alpha^{L2R}$ and $\alpha^{CMI}$) may need adjustment depending on the specific dataset.

Summary updated

Abstract

Federated Learning (FL) refers to the decentralized and privacy-preserving machine learning framework in which multiple clients collaborate (with the help of a central server) to train a global model without sharing their data. However, most existing FL methods only focus on maximizing the model's performance on the source clients' data (e.g., mobile users) without considering its generalization ability to unknown target data (e.g., a new user). In this paper, we incorporate the problem of Domain Generalization (DG) into Federated Learning to tackle the aforementioned issue. However, virtually all existing DG methods require a centralized setting where data is shared across the domains, which violates the principles of decentralized FL and hence not applicable. To this end, we propose a simple yet novel representation learning framework, namely FedSR, which enables domain generalization while still respecting the decentralized and privacy-preserving natures of this FL setting. Motivated by classical machine learning algorithms, we aim to learn a simple representation of the data for better generalization. In particular, we enforce an L2-norm regularizer on the representation and a conditional mutual information (between the representation and the data given the label) regularizer to encourage the model to only learn essential information (while ignoring spurious correlations such as the background). Furthermore, we provide theoretical connections between the above two objectives and representation alignment in domain generalization. Extensive experimental results suggest that our method significantly outperforms relevant baselines in this particular problem.
BibTeX
@inproceedings{nguyen2022fedsr,
  author = {A. Tuan Nguyen and Philip Torr and Ser-Nam Lim},
  title = {{FedSR: A Simple and Effective Domain Generalization Method for Federated Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=mrt90D00aQX},
}
Figure