Masked Autoencoders are Efficient Continual Federated Learners

Subarnaduti Paul, Lars-Joel Frey, Roshni Ramanna Kamath, Kristian Kersting, Martin Mundt

TL;DR

This paper introduces CONFEDMADE, a novel approach that leverages masked autoencoders (MADE) for efficient continual federated learning. By combining MADE's ability to learn data distributions with federated learning's distributed training and continual learning's resistance to forgetting, CONFEDMADE enables clients to learn continuously from evolving data without forgetting past knowledge, while also reducing communication costs.

What Problem Does This Paper Address?

Traditional machine learning often assumes data is independent and identically distributed (i.i.d.) and isolated. Federated learning (FL) addresses the distributed nature of data, allowing multiple clients to collaboratively train a shared model without sharing their raw data, primarily for privacy and computational reasons. However, FL often overlooks the dynamic nature of real-world data, where distributions can drift over time or individual clients might encounter new tasks. This is where continual learning (CL) comes in, aiming to prevent models from forgetting previously learned information when encountering new data or tasks.

The core challenge this paper tackles is the combination of these two paradigms: Continual Federated Learning (CFL). Existing CFL approaches often focus on supervised settings and struggle to effectively handle unsupervised representation learning in a distributed and continuously evolving environment. The paper highlights that while FL addresses data distribution across clients ("space"), and CL addresses data distribution over time ("time"), their joint consideration remains largely unexplored, especially in an unsupervised setting. The goal is to enable clients to learn continuously and collaboratively, leveraging each other's experiences indirectly, much like humans do, without suffering from catastrophic forgetting.

Methods

The proposed method, CONFEDMADE, builds upon two key ideas:

  • Masked Autoencoders (MADE): MADE is an unsupervised model originally designed for distribution estimation. Its core mechanism involves an auto-regressive masking strategy. This strategy is particularly useful because it can be adapted to learn representations that are robust to data shifts and can be integrated into a federated learning framework. The paper leverages MADE's ability to learn data distributions to build a shared understanding across clients.
  • Federated Weighted Inter-Client Transfer (FedWeIT): This existing framework for supervised CFL decomposes model parameters into "base" (generic) and "task-adaptive" (specific) components. It uses an attention mechanism to selectively transfer knowledge between clients. CONFEDMADE integrates MADE's masking strategy into this parameter decomposition framework.

The key novelty lies in how CONFEDMADE synergizes these components:

  • Unsupervised CFL: Unlike prior work that often relies on supervised signals, CONFEDMADE operates in an unsupervised manner, learning representations from data distributions.
  • Masking for Knowledge Transfer: The auto-regressive masking from MADE is used not only for distribution estimation but also to enable selective knowledge transfer between clients. This helps to reduce interference and improve the learning of shared representations.
  • Parameter Decomposition with MADE Masking: CONFEDMADE decomposes client parameters into base and task-adaptive components, similar to FedWeIT. However, it applies MADE's masking strategy to these components, further sparsifying parameters and reducing communication costs.
  • Modified Loss Function: A custom loss function is introduced to optimize the decomposed MADE client parameters, leveraging the sparsity induced by the MADE mask and minimizing interference.

Datasets

The paper evaluates CONFEDMADE on three types of dataset sequences:

  • Sequential MNIST: Comprised of 10 distinct digit classes, split into 5 consecutive tasks per client, with each task focusing on a single random digit.
  • Numerical Binary Data: Utilizes four popular binary datasets from distribution estimation literature: "RCV1", "Adult", "Connect4", and "Tretail". These are presented as sequences where each client is trained on one dataset at a time.
  • MNIST + EMNIST: A combined dataset with 36 distinct subsets (10 digits and 26 letters), where clients observe mutually exclusive samples from either digits or letters as tasks.

The paper emphasizes that labels are used only for intuition in formulating task sequences and not for the unsupervised optimization itself.

Key Insights and Ideas

  • Masked Autoencoders are Naturally Suited for CFL: The auto-regressive masking strategy in MADE, originally for distribution estimation, proves to be highly effective for unsupervised continual learning. It inherently helps in learning robust representations that can adapt to changing data distributions.
  • Synchronized Masking is Crucial in Federated Settings: When using MADE in a federated setup, independent mask sampling on each client is detrimental. Communicating a synchronized mask (or a seed to generate it) from the server to all clients is essential to avoid performance degradation due to model interference.
  • Synergy Between Masking Strategies: Combining the auto-regressive masking of MADE with the parameter decomposition and attention mechanisms of FedWeIT leads to significant improvements in continual federated learning. This synergy allows for more effective knowledge transfer and reduced forgetting.
  • Indirect Learning is Key: The paper draws inspiration from human learning, where individuals benefit from others' experiences. CONFEDMADE enables this indirect learning by allowing clients to share learned representations (task-adaptive parameters) without direct data sharing, fostering collaborative and continuous learning.
  • Communication Efficiency through Masking: By leveraging MADE's masking, CONFEDMADE can communicate only non-zero parameters, significantly reducing communication overhead compared to traditional FL approaches, especially when combined with parameter decomposition.

Key Results

  • Mitigated Forgetting: CONFEDMADE consistently reduces forgetting across various datasets and learning settings. For example, in the MNIST dataset, it achieved a forgetting value of 8.32 ± 0.76, significantly lower than other baselines.
  • Improved Negative Log-Likelihood (NLL): The model achieves competitive or superior NLL scores compared to other CFL approaches, indicating better reconstruction of data distributions. For instance, on the Binary dataset, CONFEDMADE achieved an average NLL of 59.15 ± 0.67.
  • Reduced Communication Costs: By masking parameters, CONFEDMADE reduces the communication cost by approximately 50% compared to FedWeIT-MADE, with minimal impact on performance.
  • Effective Inter-Client Knowledge Transfer: Heatmaps illustrating attention values (α) show that CONFEDMADE clients effectively leverage shared knowledge when there is overlap in observed data subsets, leading to better learning and reduced forgetting.

Limitations and Future Work

  • Computational Cost: While CONFEDMADE reduces communication, the computational cost on clients can still be significant, especially with complex architectures.
  • Hyperparameter Tuning: The performance of CONFEDMADE relies on careful tuning of hyperparameters, such as the masking ratio and regularization weights.
  • Future Work: The authors suggest scaling CONFEDMADE to more complex architectures like Transformers, which have shown promise with masked autoencoding. They also propose investigating multi-modal learning scenarios where clients might observe different types of data. Further research could explore more sophisticated ways to integrate attention mechanisms with MADE's masking for even more selective knowledge transfer.

Summary updated

Abstract

Machine learning is typically framed from a perspective of i.i.d., and more importantly, isolated data. In parts, federated learning lifts this assumption, as it sets out to solve the real-world challenge of collaboratively learning a shared model from data distributed across clients. However, motivated primarily by privacy and computational constraints, the fact that data may change, distributions drift, or even tasks advance individually on clients, is seldom taken into account. The field of continual learning addresses this separate challenge and first steps have recently been taken to leverage synergies in distributed supervised settings, in which several clients learn to solve changing classification tasks over time without forgetting previously seen ones. Motivated by these prior works, we posit that such federated continual learning should be grounded in unsupervised learning of representations that are shared across clients; in the loose spirit of how humans can indirectly leverage others' experience without exposure to a specific task. For this purpose, we demonstrate that masked autoencoders for distribution estimation are particularly amenable to this setup. Specifically, their masking strategy can be seamlessly integrated with task attention mechanisms to enable selective knowledge transfer between clients. We empirically corroborate the latter statement through several continual federated scenarios on both image and binary datasets.
BibTeX
@inproceedings{paul2024masked,
  author = {Subarnaduti Paul and Lars-Joel Frey and Roshni Ramanna Kamath and Kristian Kersting and Martin Mundt},
  title = {{Masked Autoencoders are Efficient Continual Federated Learners}},
  booktitle = {CoLLAs},
  year = {2024},
}
Figure