Exploring Federated Unlearning: Review, Comparison, and Insights

Yang Zhao, Jiaxi Yang, Yiling Tao, Lixu Wang, Xiaoxiao Li, Dusit Niyato, H. Vincent Poor

TL;DR

This paper dives into the emerging field of federated unlearning, which allows for the selective removal of data from trained machine learning models in a privacy-preserving manner. It provides a comprehensive review and comparison of existing methods, highlighting the critical trade-offs between privacy, accuracy, and efficiency, and introduces a new benchmark framework, OPENFEDERATEDUNLEARNING, to standardize evaluation.

What Problem Does This Paper Address?

Federated Learning (FL) trains models across decentralized devices without sharing raw data, offering a privacy advantage. However, FL models can still inadvertently reveal sensitive information through gradient updates, making them vulnerable to attacks like membership inference. Furthermore, evolving privacy regulations like GDPR grant users the "right to be forgotten," necessitating mechanisms to remove specific data from already trained models. Federated unlearning aims to address this by enabling the selective erasure of data's influence from a federated model after training, without requiring a full retraining from scratch, which is computationally prohibitive. The core challenge lies in balancing the conflicting objectives of effectively removing data (privacy), maintaining model performance (accuracy), and doing so efficiently.

Methods

The paper categorizes federated unlearning approaches into three main types:

  • Training-Based Approaches: These involve retraining the model with the remaining dataset after removing specific data. While guaranteeing complete erasure, they are computationally expensive.
  • Tuning-Based Approaches: These methods adjust the parameters of a pre-trained model to selectively forget data influence. They are generally more efficient than retraining but require careful implementation to ensure complete removal.
  • Model-Based Approaches: These modify the model's structure directly to eliminate the influence of specific data, often effective for class-level unlearning.

The paper also discusses different granularities of unlearning: class-level, client-level, and sample-level. A key contribution is the proposal of the OPENFEDERATEDUNLEARNING framework, a unified benchmark designed to evaluate federated unlearning methods using classic baselines and diverse performance metrics.

Datasets

The experimental evaluation in the paper uses the following standard datasets:

  • MNIST
  • CIFAR-10
  • SVHN

The paper also introduces the OPENFEDERATEDUNLEARNING framework as a benchmark, which incorporates classic baselines and diverse performance metrics for evaluating federated unlearning approaches.

Key Insights and Ideas

  • No Silver Bullet: The paper strongly emphasizes that no single federated unlearning method is universally optimal. The choice of method depends heavily on the specific application requirements, dataset characteristics, and the desired balance between privacy, accuracy, and efficiency.
  • The Accuracy-Efficiency-Privacy Trilemma: A central theme is the inherent trade-off between these three critical objectives. More complex models (higher accuracy) often require more effort to unlearn (lower efficiency). Similarly, achieving perfect data erasure (high privacy) might come at the cost of accuracy or efficiency.
  • The Importance of Benchmarking: The authors highlight the need for standardized evaluation. Their proposed OPENFEDERATEDUNLEARNING framework aims to address this by providing a consistent platform for comparing different methods.
  • Beyond Simple Retraining: While retraining is the most straightforward way to "forget" data, its computational cost makes it impractical for large-scale federated systems. This drives the need for more sophisticated unlearning techniques.
  • Granularity Matters: The effectiveness and challenges of unlearning vary significantly depending on whether one is trying to remove an entire class, a specific client's data, or individual data points.

Key Results

The experimental results, presented in Table II and III, show:

  • Client-Level Unlearning: FedEraser demonstrated strong performance in approximating the retrained model across datasets like MNIST and SVHN, indicating good model integrity. However, methods like Fine-Tuning and C2T, while achieving high test accuracies, sometimes struggled with complete data erasure, as indicated by higher backdoor attack (BA) and membership inference attack (MIA) rates.
  • Class-Level Unlearning: On CIFAR-10, class-level unlearning effectively reduced the accuracy of the targeted class while maintaining reasonable performance on other classes. Privacy metrics showed moderate improvements, but an increase in EM (Erroneous Memory) indicated that unlearning might not be fully complete.
  • Trade-offs in Action: For complex datasets like CIFAR-10, methods that preserved model accuracy (Fine-Tuning, C2T) sometimes showed less effective data erasure. Conversely, methods focusing on erasure might lead to accuracy degradation.
  • Efficiency Gains: Tuning-based approaches generally offered significant speedups compared to full retraining. For instance, FedEraser showed a notable reduction in unlearning time.

Limitations and Future Work

Limitations:

  • Dataset Complexity: The paper notes that unlearning becomes more challenging with complex datasets, where achieving both complete erasure and high accuracy simultaneously is difficult.
  • No Universal Solution: As mentioned, the lack of a single method that excels across all metrics and scenarios is a significant limitation for practitioners.

Future Work:

  • Incentive Mechanism Design: Developing mechanisms to motivate user participation in unlearning while ensuring system efficiency and minimal accuracy loss.
  • Federated Unlearning for LLMs: Addressing the unique challenges of unlearning data from massive Large Language Models, where data is distributed across billions of parameters.
  • Green Federated Unlearning: Investigating methods to reduce the potentially higher carbon footprint of federated unlearning compared to centralized approaches, focusing on reducing communication rounds and computational needs.
  • Developing More Robust Benchmarks: Continuing to refine evaluation metrics and frameworks to better capture the nuances of federated unlearning.

Summary updated

Abstract

The increasing demand for privacy-preserving machine learning has spurred interest in federated unlearning, which enables the selective removal of data from models trained in federated systems. However, developing federated unlearning methods presents challenges, particularly in balancing three often conflicting objectives: privacy, accuracy, and efficiency. This paper provides a comprehensive analysis of existing federated unlearning approaches, examining their algorithmic efficiency, impact on model accuracy, and effectiveness in preserving privacy. We discuss key trade-offs among these dimensions and highlight their implications for practical applications across various domains. Additionally, we propose the OPENFEDERATE-DUNLEARNING framework, a unified benchmark for evaluating federated unlearning methods, incorporating classic baselines and diverse performance metrics. Our findings aim to guide practitioners in navigating the complex interplay of these objectives, offering insights to achieve effective and efficient federated unlearning. Finally, we outline directions for future research to further advance the state of federated unlearning techniques.
BibTeX
@article{zhao2025exploring,
  author = {Yang Zhao and Jiaxi Yang and Yiling Tao and Lixu Wang and Xiaoxiao Li and Dusit Niyato and H. Vincent Poor},
  title = {{Exploring Federated Unlearning: Review, Comparison, and Insights}},
  journal = {IEEE Network},
  year = {2025},
  doi = {10.1109/mnet.2025.3571462},
}
Figure