Federated Impression for Learning with Distributed Heterogeneous Data

Atrin Arya, Sana Ayromlou, Armin Saadat, Purang Abolmaesumi, Xiaoxiao Li

Abstract

Federated Learning (FL) often struggles with data heterogeneity, where non-identically and independently distributed (non-IID) data across clients leads to "catastrophic forgetting." This phenomenon occurs when local updates overwrite global knowledge, causing model drift and poor convergence. This paper introduces FedImpres (Federated Impression), a framework that mitigates forgetting by synthesizing a "federated impression" of global data at the server. By sharing these synthetic samples with clients to serve as a global anchor during local training, FedImpres achieves significant performance gains, including up to a 20% accuracy improvement in highly heterogeneous medical imaging scenarios.

Introduction

Federated Learning (FL) has emerged as a vital paradigm for training machine learning models across decentralized institutions, such as hospitals, without compromising data privacy. By keeping raw data local and only sharing model weights, FL addresses many regulatory and ethical concerns. However, the practical application of FL is frequently hindered by data heterogeneity. In medical contexts, this manifests as label imbalance (e.g., varying prevalence of specific diseases across clinics) and domain shift (e.g., differences in imaging hardware or protocols).

The primary consequence of this heterogeneity is catastrophic forgetting. When a client receives a global model and optimizes it against a unique, biased local dataset, the weights are modified so aggressively that the model loses the generalized knowledge acquired from other participants. This leads to a model that performs well on local data but fails to generalize to the broader federation, ultimately stalling convergence.

Methodology: The FedImpres Framework

FedImpres addresses catastrophic forgetting through a two-stage process that utilizes data-level regularization rather than traditional weight-based constraints.

1. Server-Side: Generating the "Federated Impression"

After the server aggregates weights from participating clients using standard algorithms like FedAvg, it synthesizes a prototypical dataset known as the "Federated Impression." Unlike generative approaches that require training a GAN, FedImpres uses a data-free knowledge distillation technique.

The server performs pixel-wise optimization to create images that the current global model recognizes as representative of the global distribution. To ensure these images are truly "information-preserving," the authors employ a model gradient-based constraint solved via the Alternating Direction Method of Multipliers (ADMM). This ensures that a model trained on this synthetic data would behave similarly to the aggregated global model, effectively capturing the collective knowledge of the federation without accessing raw client data.

2. Client-Side: Forgetting-Penalized Training

In each communication round, the server distributes both the global model weights and the synthetic impression dataset to the clients. During local optimization, clients minimize a joint loss function:

  • Local Task Loss: Standard cross-entropy loss on the client's private, heterogeneous data.
  • Regularization Loss: A penalty based on the synthetic global data.

By including the synthetic data in the local training loop, the impression acts as a "memory" of the global distribution. This prevents the local update from drifting too far toward the local bias, ensuring the model retains its ability to classify data from other domains and classes.

Experimental Evaluation

The researchers evaluated FedImpres against two challenging medical imaging benchmarks designed to simulate real-world heterogeneity:

  • BloodMNIST: A dataset of microscopic peripheral blood cell images. Heterogeneity was simulated using Latent Dirichlet Allocation (LDA) to create "moderate" and "severe" label imbalances.
  • Retina Dataset: A multi-site dataset for glaucoma detection. This benchmark features natural heterogeneity, including both label imbalance and significant domain shifts caused by different acquisition settings across four distinct clinical sites.

Key Results and Insights

FedImpres demonstrated substantial improvements over baseline methods such as FedAvg and FedProx, as well as specialized algorithms like FedCurv and VHL:

  • Significant Accuracy Gains: On the BloodMNIST dataset with severe label imbalance, FedImpres improved classification accuracy by up to 20% compared to standard FedAvg.
  • Enhanced Stability: While standard FL methods often show a sharp drop in accuracy on "other" clients' data during local training (a clear sign of forgetting), FedImpres maintained stable performance across the entire federation throughout the training process.
  • Adaptive Regularization: Unlike static synthetic data methods, FedImpres regenerates the impression in every round. This allows the synthetic data to evolve alongside the global model, capturing increasingly complex features as the model matures.
  • Privacy and Flexibility: The synthesis process relies entirely on aggregated weights, maintaining strict privacy. Furthermore, while the method performs best when initialized with a few unlabeled proxy images, it remains effective even when starting from random noise.

Conclusion and Future Work

FedImpres offers a robust solution to the challenge of non-IID data in Federated Learning by treating the problem at the data level. By providing clients with a synthetic "impression" of the global data, the framework effectively anchors local training and prevents the catastrophic forgetting that plagues traditional FL.

While the method introduces some computational overhead due to the per-round synthesis at the server, it avoids the complexity of training separate generative models. Future research may focus on reducing this overhead further and exploring the application of FedImpres to 3D medical volumes and other complex data modalities.

Summary updated

BibTeX
@inproceedings{arya2024federated,
  author = {Atrin Arya and Sana Ayromlou and Armin Saadat and Purang Abolmaesumi and Xiaoxiao Li},
  title = {{Federated Impression for Learning with Distributed Heterogeneous Data}},
  booktitle = {ISIC/iMIMIC/EARTH/DeCaF @ MICCAI},
  year = {2024},
  doi = {10.1007/978-3-031-77610-6\_20},
}
Figure