FRAug: Tackling Federated Learning with Non-IID Features via Representation Augmentation

Haokun Chen, Ahmed Frikha, Denis Krompass, Jindong Gu, Volker Tresp

TL;DR

FRAug (Federated Representation Augmentation) is a new approach to Federated Learning that solves the problem of "feature shift"—where different clients have data that looks different (e.g., medical images from different scanners) despite sharing the same labels. By generating synthetic, personalized data in a low-dimensional embedding space, FRAug helps local models generalize better and significantly outperforms current state-of-the-art methods like FedBN and PartialFed.

What Problem Does This Paper Address?

In Federated Learning (FL), multiple participants (clients) train a model together without ever sharing their raw data. While great for privacy, FL struggles with non-IID data—data that isn't "Independent and Identically Distributed."

Most research focuses on label skew (e.g., Client A has only cat pictures, Client B has only dogs). However, this paper tackles the under-explored challenge of feature distribution shift (or covariate shift). Imagine several hospitals trying to detect the same type of cancer, but each uses a different brand of MRI scanner. The "features" (the visual style of the scans) differ wildly between hospitals, even though the "labels" (cancer vs. no cancer) are the same. This heterogeneity causes local models to drift away from a global consensus, leading to poor performance when the model encounters data from a slightly different "domain."

Methods

The authors propose FRAug, which uses data augmentation not in the raw pixel space, but in the embedding space (the internal mathematical representation the model creates).

The process involves three main components:

  1. Shared Generator: A global model trained to capture "client-agnostic" knowledge. It learns to produce generic feature embeddings that represent the core concepts of the classes across all clients.
  2. Representation Transformation Network (RTNet): Each client trains a local RTNet. This network takes the generic embeddings from the shared generator and transforms them into "client-specific" embeddings that match the local data's style.
  3. Augmentation: During local training, the client uses its real data plus these synthetic, personalized embeddings to train its classification head.
Overview of the FRAug architecture showing how the generator and RTNet create synthetic embeddings to augment local training.

By using Maximum Mean Discrepancy (MMD) loss, the system ensures that the generator stays "agnostic" while the RTNet successfully "personalizes" the data, effectively acting as a regularizer that prevents the model from overfitting to its specific local "look."

Datasets

The authors evaluated FRAug on four diverse benchmarks:

  • OfficeHome: Images of everyday objects in four styles: Art, Clipart, Product, and Real-World.
  • PACS: A classic domain generalization benchmark featuring Photos, Art, Cartoons, and Sketches.
  • Digits: A mix of four digit datasets (MNIST, MNIST-M, SVHN, and USPS), each with very different backgrounds and textures.
  • RxRx1: A real-world medical dataset containing fluorescent microscopy images of four different cell types. This dataset is particularly challenging due to the high degree of visual heterogeneity between cell batches.
Figure 2: Example images of different cell types, i.e., lo- cal data from different clients, in RxRx1 dataset. Strong feature space heterogeneity can be observed between image appearance. Best viewed in color.

Key Insights and Ideas

  • Embedding Space > Pixel Space: Generating synthetic images (pixels) is computationally expensive and poses higher privacy risks. Generating embeddings is faster, more efficient for edge devices, and keeps the augmentation closer to the decision-making part of the model.
  • The "Consensus vs. Personalization" Balance: The shared generator acts as the "global wisdom," while the RTNet acts as the "local expert." This division of labor allows the model to learn universal features while still respecting local data variations.
  • Finetuning the Head is Key: A motivational study in the paper revealed that even if a feature extractor is trained on limited data, the final classification "head" can be significantly improved if it's exposed to a wider variety of synthetic representations.

Key Results

FRAug consistently outperformed all baselines, including standard FedAvg and specialized methods like FedBN.

  • Benchmark Superiority: On the Digits dataset, FRAug achieved an average accuracy of 89.59%, a substantial 2.4% lead over FedBN.
  • Medical Scalability: On the complex RxRx1 dataset, FRAug maintained its lead as the number of clients increased (from 8 to 16), showing it can handle the scaling demands of real-world medical collaborations.
  • Faster Convergence: As shown in the convergence plots, FRAug reaches higher accuracy levels much faster than FedAvg or FedBN, often showing distinct gains within the first 50 communication rounds.
Figure 3: Convergence analysis of FedAvg, FedBN and FRAug on (a) OfficeHome and (b) Digits benchmarks.

Limitations and Future Work

While FRAug is highly effective, the authors note a few areas for further exploration:

  • Computational Overhead: Although the generator and RTNet are small (adding only ~2-3% more parameters), they still represent an extra step in the training pipeline.
  • Privacy Guarantees: While embedding-based augmentation is inherently more private than sharing images, the paper does not provide a formal Differential Privacy analysis, which might be required for highly sensitive medical applications.
  • Future Directions: The authors suggest that FRAug could be combined with other defensive strategies to improve robustness against adversarial attacks in decentralized settings.

Summary updated

Abstract

Federated Learning (FL) is a decentralized learning paradigm, in which multiple clients collaboratively train deep learning models without centralizing their local data, and hence preserve data privacy. Real-world applications usually involve a distribution shift across the datasets of the different clients, which hurts the generalization ability of the clients to unseen samples from their respective data distributions. In this work, we address the recently proposed feature shift problem where the clients have different feature distributions, while the label distribution is the same. We propose Federated Representation Augmentation (FRAug) to tackle this practical and challenging problem. Our approach generates synthetic client-specific samples in the embedding space to augment the usually small client datasets. For that, we train a shared generative model to fuse the clients knowledge learned from their different feature distributions. This generator synthesizes client-agnostic embeddings, which are then locally transformed into client-specific embeddings by Representation Transformation Networks (RTNets). By transferring knowledge across the clients, the generated embeddings act as a regularizer for the client models and reduce overfitting to the local original datasets, hence improving generalization. Our empirical evaluation on public benchmarks and a real-world medical dataset demonstrates the effectiveness of the proposed method, which substantially outperforms the current state-of-the-art FL methods for non-IID features, including PartialFed and FedBN.
BibTeX
@inproceedings{chen2022fraug,
  author = {Haokun Chen and Ahmed Frikha and Denis Krompass and Jindong Gu and Volker Tresp},
  title = {{FRAug: Tackling Federated Learning with Non-IID Features via Representation Augmentation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2022},
}
Figure