Abstract
Federated Learning (FL) enables collaborative model training across distributed devices while keeping data localized. However, deep neural networks often unintentionally memorize specific training instances, posing a significant privacy risk when users exercise their "right to be forgotten." Traditional Federated Unlearning (FU) methods are typically post-hoc and computationally expensive. We introduce ToFU (Transformation-guided Federated Unlearning), a paradigm shift that integrates unlearning capabilities directly into the training process. By applying sample-dependent transformations and progressive training strategies, ToFU proactively limits memorization. Our results demonstrate that ToFU significantly reduces unlearning time and enhances privacy while maintaining or even improving model utility across standard benchmarks.
Introduction
Deep neural networks are notoriously "leaky," often memorizing specific training samples rather than solely learning generalizable patterns. In Federated Learning environments, where models are trained on sensitive data from phones or medical servers, this memorization creates a substantial privacy bottleneck. When a user requests data deletion, the system must remove that user's influence from the global model without access to the original raw data.
Current FU methods usually attempt to "scrub" the model after training is complete. This post-hoc approach is difficult because user knowledge is deeply embedded in the model weights. ToFU addresses this by moving from a "fix-it-later" strategy to a "design-to-forget" training methodology, ensuring the model is inherently easier to unlearn from the start.
Methodology
ToFU incorporates unlearning capabilities into the collaborative training process through three core strategies:
- Sample-Dependent Transformation: ToFU monitors the loss for each sample. Samples with very low loss—indicating potential memorization—are subjected to higher-intensity transformations (e.g., masking or augmentations). This forces the model to focus on invariant features rather than specific pixel arrangements.
- Progressive Training: The intensity of these transformations increases linearly across communication rounds. This allows the model to learn basic features early on before being challenged with increasingly complex, transformed data.
- Transformation-Invariant Regularizer: A loss term based on Kullback–Leibler (KL) divergence encourages the model's latent representations of transformed images to remain close to their original counterparts, ensuring the model learns the underlying object identity.
As shown in the figure above, there is a strong positive correlation between the number of transformations applied during training and the resulting unlearning performance, validating the core premise of the transformation-guided approach.
Experimental Evaluation
We evaluated ToFU using standard image classification benchmarks (CIFAR-10 and CIFAR-100) and the MUFAC (Machine Unlearning Facial Classification) dataset, which focuses on sensitive facial recognition tasks.
Key Results
- Efficiency: ToFU reduces unlearning time by 24% to 28% compared to post-hoc baselines. Because the model is trained to be "forget-friendly," the final unlearning step is reduced to a lightweight fine-tuning operation.
- Privacy: The method significantly lowers the success rate of Membership Inference Attacks (MIA). ToFU achieves a level of distributional similarity to "exact unlearning" (retraining from scratch) that surpasses existing methods.
- Utility: Unlike many privacy-preserving techniques that degrade performance, ToFU maintains high accuracy. On CIFAR-10, ToFU achieved an overall performance score of 0.6712, outperforming the best baseline (0.6565).
As illustrated in the validation curves, ToFU consistently achieves higher accuracy than naive federated learning throughout the training process, demonstrating that its privacy-focused transformations do not come at the cost of model utility.
Discussion and Conclusion
ToFU demonstrates that the primary bottleneck in federated unlearning is not the unlearning algorithm itself, but the initial training process. By proactively limiting memorization, we make the "scrubbing" process nearly trivial. While the initial training phase involves slightly more local computation due to transformations and KL-divergence calculations, this is a worthwhile trade-off for the substantial gains in privacy and unlearning efficiency.
Future work will focus on refining transformation strategies for highly specialized domains like facial recognition, where we observed a marginal decrease in accuracy on the MUFAC dataset. Additionally, automating the selection of transformation families via methods like RandAugment could further streamline the ToFU pipeline.
Summary updated