Abstract
As Federated Learning (FL) transitions from research settings to dynamic real-world applications, models must adapt to both acquiring new skills and "forgetting" specific data for privacy or efficiency. This paper introduces Federated Editable Learning (FEL), a unified framework that categorizes state-of-the-art techniques in knowledge augmentation (continual learning) and knowledge removal (unlearning), while emphasizing the critical role of verification and auditing in the model lifecycle.
Introduction
Standard Federated Learning typically assumes a static environment with fixed data distributions and tasks. However, real-world deployments face two primary "dynamics":
- Data Dynamics: Local clients continuously receive new data or tasks. This often leads to catastrophic forgetting, where the global model loses previously acquired knowledge while attempting to integrate new information.
- Device Dynamics: Clients frequently join or leave the network. Privacy regulations, such as the GDPR, grant users the "right to be forgotten," necessitating methods to remove a specific client's influence from the global model without the prohibitive cost of retraining from scratch—a process known as Federated Unlearning (FU).
This paper addresses the gap in current literature by providing a holistic view of "Knowledge Editing"—the ability to precisely add or subtract information from a distributed model.
The Federated Editable Learning (FEL) Paradigm
The authors propose the Federated Editable Learning (FEL) framework, which structures the field into three main pillars: knowledge increase, knowledge removal, and verification.
1. Federated Increasing Learning (FIL)
FIL focuses on expanding the model's knowledge base through two primary sub-fields:
- Federated Continual Learning (FCL): Employs techniques to mitigate catastrophic forgetting, including regularization-based approaches (penalizing changes to critical weights) and exemplar-based methods (retaining a subset of old samples).
- Federated Few-shot Learning (FFsL): Utilizes meta-learning and adversarial training to enable models to generalize to new tasks using very limited local data.
2. Federated Unlearning (FU)
FU focuses on the precise removal of specific knowledge, categorized by the "exactness" of the operation:
- Exact Unlearning: Involves retraining or using modular architectures like LegoNet, which employs isolated "adapters" that can be swapped or deleted to remove specific influences.
- Approximate Unlearning: Offers faster "scrubbing" methods, such as Gradient Recovery (reversing model states using historical updates) or Channel Pruning (identifying and removing neurons associated with the forgotten data).
3. Verification & Auditing
The authors argue that a model "edit" is only valid if it can be verified. They review methods such as Trusted Execution Environments (TEEs) and Proof of Learning to ensure honest training, alongside Membership Inference Attacks to confirm that "unlearned" data has indeed been removed.
Key Insights and Future Directions
The survey highlights several critical perspectives for the future of FL:
- The Capacity Limit: Models have a finite "knowledge upper limit." Unlearning serves as a management tool to remove obsolete data, freeing capacity for new, relevant knowledge.
- Knowledge Disentanglement: Currently, knowledge in neural networks is highly entangled. Removing specific data points often inadvertently degrades performance on unrelated tasks. Moving toward modular, "Lego-like" architectures is a promising path for efficient editing.
- Trustworthy Communities: Integrating Blockchain with FL could create a transparent, tamper-proof ledger of all model edits, ensuring accountability in decentralized AI systems.
Conclusion
Federated Editable Learning represents a shift toward more flexible and compliant distributed AI. By unifying learning and unlearning under a single framework of "Knowledge Editing," researchers can better address the complexities of dynamic real-world data and privacy requirements.
Summary updated