Federated Learning (FL) has traditionally operated under a "static world" assumption, where data distributions and tasks are fixed at the start of training. However, real-world environments are inherently dynamic. This paper provides a comprehensive roadmap for evolving FL systems into sustainable ecosystems capable of incorporating "new knowledge"—including novel features, tasks, models, and algorithms—without the prohibitive costs of retraining from scratch.
The Challenge of Static Federated Learning
While FL excels at privacy-preserving decentralized learning, existing frameworks often struggle when environments shift. For instance, healthcare systems must adapt to new diseases, and autonomous vehicles must handle changing road conditions or sensor upgrades. Discarding a large-scale FL system due to these changes is inefficient. The authors propose a shift toward "sustainable evolution," focusing on how to integrate new information while preserving previously acquired knowledge.
A Taxonomy for New Knowledge
The paper establishes a formal taxonomy for categorizing new knowledge in FL across four key variables:
- New Features ($\mathcal{X}$): Handled via Federated Domain Generalization (FDG) for robustness to unseen data and Federated Domain Adaptation (FDA) to bridge distribution gaps.
- New Tasks ($\mathcal{Y}$): Addressed through Task-Personalized FL and Self-Supervised FL (SSFL) to prepare models for future requirements.
- New Tasks + Features: This is the domain of Federated Continual Learning (FCL), which utilizes replay-based or architecture-based techniques to prevent "catastrophic forgetting" as clients learn new classes over time.
- New Models ($\mathcal{M}$) and Algorithms ($\mathcal{A}$): Discusses architecture transfer (e.g., moving from ResNets to Transformers) and the challenges of switching aggregation protocols mid-stream.
Key Insights for System Sustainability
- The Lifespan Metric: The viability of an FL system is tied to client retention. If adaptation to new demands is too slow, clients withdraw, effectively ending the system's lifespan.
- Client Categorization: Effective knowledge integration requires managing three client types: $C_{cold}$ (old knowledge only), $C_{new}$ (new knowledge only), and $C_{mix}$ (both).
- The Foundation Model Shift: A major trend is the transition toward incorporating Foundation Models (FMs) and LLMs. Future FL will likely involve distilling knowledge from massive server-side models into local client models rather than training from scratch.
State-of-the-Art Performance
The paper highlights significant progress across various sub-fields using benchmark datasets like PACS, CIFAR, and ImageNet. For example, recent Federated Continual Learning methods on ImageNet have achieved 62.1% accuracy with forgetting rates as low as 0.2%, demonstrating that models can learn new tasks without erasing prior knowledge.
Future Research Directions
Despite recent advances, several challenges remain:
- Efficiency Gap: Future work must optimize the communication and computation overhead associated with incorporating updates.
- Optimal Switching: Determining the ideal timing for transitioning between algorithms or models is still an open question.
- Security Risks: New knowledge introduces novel attack surfaces, where malicious clients might disguise poisoning or backdoor attacks as legitimate updates.
- Task Complexity: Moving beyond simple classification toward multi-modal and complex reasoning tasks remains a critical frontier for the field.
Summary updated