FedMeS: Personalized Federated Continual Learning Leveraging Local Memory

Jin Xie, Chenqing Zhu, Songze LiarXiv 2024 · OpenReview

Abstract

We focus on the problem of Personalized Federated Continual Learning (PFCL): a group of distributed clients, each with a sequence of local tasks on arbitrary data distributions, collaborate through a central server to train a personalized model at each client, with the model expected to achieve good performance on all local tasks. We propose a novel PFCL framework called Federated Memory Strengthening (FedMeS) to address the challenges of client drift and catastrophic forgetting. In FedMeS, each client stores samples from previous tasks using a small amount of local memory and leverages this information to both: 1) calibrate gradient updates in the training process; and 2) perform KNN-based Gaussian inference to facilitate personalization. FedMeS is designed to be task-oblivious, such that the same inference process is applied to samples from all tasks to achieve good performance. FedMeS is analyzed theoretically and evaluated experimentally, showing it outperforms all baselines in average accuracy and forgetting rate over various combinations of datasets, task distributions, and client numbers.

Introduction

In real-world machine learning applications, such as a network of factories using Federated Learning (FL) for defect detection, models face two primary challenges: client drift and catastrophic forgetting. Client drift occurs because each factory produces different items, leading to non-IID (non-independent and identically distributed) data where a single global model fails to serve all participants effectively. Catastrophic forgetting happens as production lines evolve; learning new tasks often overwrites the knowledge required for previous ones.

This paper introduces FedMeS (Federated Memory Strengthening), a framework specifically designed for Personalized Federated Continual Learning (PFCL). FedMeS aims to develop personalized models that maintain high performance across a sequence of local tasks by leveraging a small local memory buffer at each client.

Methodology

The core innovation of FedMeS is the strategic use of a local memory buffer containing exemplars from previous tasks to enhance both the training and inference stages.

Training with Gradient Calibration

During local training, FedMeS mitigates forgetting by ensuring that updates for new tasks do not conflict with prior knowledge. It calculates the inner product between the current task's gradient and the gradient derived from the memory buffer. If a conflict is detected (indicated by a negative inner product), a gradient correction step projects the update into a "safe" direction that does not harm previous performance. Additionally, FedMeS employs dynamic personalization by adjusting the regularization parameter ($\lambda$) based on local loss, allowing models to deviate from the global average when necessary to better fit local data.

Task-Oblivious KNN Inference

A distinguishing feature of FedMeS is its inference stage. Instead of relying solely on neural network outputs, it performs KNN-based Gaussian inference using the stored memory. By combining model predictions with similarity-based predictions from the buffer, FedMeS achieves superior accuracy. Crucially, this process is task-oblivious, meaning the model does not require a "task ID" at test time, making it highly practical for real-world deployment where the source of a data point might be unknown.

Overview of the FedMeS framework showing local training and inference processes.

Experimental Evaluation

The researchers evaluated FedMeS against several benchmarks, including Split CIFAR-100, Split EMINIST, Split CORe50, and Split MiniImageNet. They also introduced Fusion Tasks-40, a cross-domain benchmark combining 200,000 images from multiple sources into 40 non-IID tasks.

Key Results

FedMeS consistently outperformed baselines such as FedAvg, Ditto, and FedWeIT in both accuracy (Acc) and forgetting rate (FR). For instance, on Split MiniImageNet, it achieved up to a 102% performance boost over the state-of-the-art FedWeIT. On the Split CORe50 dataset, the forgetting rate was reduced by 4 to 6 times compared to other methods.

Experimental results comparing FedMeS against baselines across multiple datasets.

The framework also demonstrated high communication efficiency and scalability, maintaining performance as the number of clients increased from 10 to 20, showing better robustness than isolation-based methods.

Conclusion

FedMeS effectively addresses the dual challenges of client drift and catastrophic forgetting in PFCL. By repurposing local memory for both gradient calibration and task-oblivious inference, it provides a robust and efficient solution for distributed continual learning. While it introduces a small memory overhead, the performance gains and task-oblivious nature make it a compelling choice for real-world federated systems. Future work may explore synthetic memory to further enhance privacy and extend the framework to domains beyond image classification, such as Natural Language Processing.

Summary updated

Abstract

We focus on the problem of Personalized Federated Continual Learning (PFCL): a group of distributed clients, each with a sequence of local tasks on arbitrary data distributions, collaborate through a central server to train a personalized model at each client, with the model expected to achieve good performance on all local tasks. We propose a novel PFCL framework called Federated Memory Strengthening FedMeS to address the challenges of client drift and catastrophic forgetting. In FedMeS, each client stores samples from previous tasks using a small amount of local memory, and leverages this information to both 1) calibrate gradient updates in training process; and 2) perform KNN-based Gaussian inference to facilitate personalization. FedMeS is designed to be task-oblivious, such that the same inference process is applied to samples from all tasks to achieve good performance. FedMeS is analyzed theoretically and evaluated experimentally. It is shown to outperform all baselines in average accuracy and forgetting rate, over various combinations of datasets, task distributions, and client numbers.
BibTeX
@misc{xie2024fedmes,
  author = {Jin Xie and Chenqing Zhu and Songze Li},
  title = {{FedMeS: Personalized Federated Continual Learning Leveraging Local Memory}},
  year = {2024},
  eprint = {2404.12710},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.12710},
}
Figure