When a machine learning model trains on user data, it doesn't just store that data in a database; it weaves the information into millions or billions of numerical weights. Under modern privacy frameworks like the European Union's GDPR, users possess the "Right to be Forgotten," meaning they can demand that their data be completely expunged from these models.
In a centralized setting, the gold standard for data removal is simple: delete the user's data from the server and retrain the model from scratch. However, as Baracaldo notes, modern applications increasingly rely on Federated Learning (FL), where data remains siloed on decentralized edge devices, and only model updates (gradients) are aggregated by a central server.
If a client requests data revocation in a federated network, a naive solution would be to simply stop including that client in future training rounds. However, as demonstrated by Nguyen et al., models implicitly retain features, patterns, and even backdoors from past participants indefinitely. Fulfilling privacy mandates requires active intervention to completely purge this historical influence without incurring the prohibitive computational cost of rebuilding the global model from scratch.
The Challenge of Entanglement Without Raw Data
Why is federated unlearning uniquely difficult? In centralized machine unlearning, algorithms can directly manipulate or evaluate the original training samples to guide the forgetting process. But as Shaik et al. point out, a federated server never has access to raw data. It only observes aggregated parameter trajectories.
Furthermore, the updates from the target client are deeply entangled with the updates of all other participants through rounds of global averaging. Directly altering a model to forget one client's data risks destroying the shared knowledge contributed by everyone else. According to Jeong et al., this distributed and interactive nature introduces client stochasticity and data heterogeneity, meaning centralized unlearning techniques cannot be directly applied.
To solve this, researchers have developed three primary intuitions for how a distributed model can "forget": rolling back parameter history, actively destroying features via localized perturbation, and collaboratively repairing the resulting model.
Mechanism 1: Rolling Back and Calibrating History
If the central server cannot look at the raw data, it can look at the history of how the model's weights changed over time. The intuition here is simple: if we know exactly how a client's updates pushed the model's parameters in a certain direction, we can subtract that specific trajectory.
Global Model State (Round t) <-- Contains Entangled Updates
│
▼
Subtract Target Client's Historical Footprint
│
▼
Calibrate & Roll Back to Clean State
As explored by Xu et al., unlearning can be achieved by leveraging historical parameter updates to subtract the specific contributions of a single node. Rather than saving every single model state across thousands of rounds—which would cause storage requirements to skyrocket—Jiang et al. show that the server can selectively retain a strategic subset of historical global models and client updates that exhibit high alignment or significant divergence.
Once the historical state is isolated, the model can be rolled back. Yuan et al. demonstrate that calibrating historical updates after rolling back a global model state allows systems to efficiently erase specific user contributions and recover model performance up to 7x faster than full retraining. For modular architectures, Gu shows that this weight manipulation can even be performed near-instantly by using linear algebra to add or subtract weights from a lightweight auxiliary module trained alongside the main model.
Mechanism 2: Active Feature Perturbation
Instead of rewinding the clock, an alternative approach forces the model to actively "unlearn" the target data by running its optimization process in reverse.
If gradient descent minimizes loss to help a model learn, Gradient Ascent maximizes loss on the target data to force the model to forget it. Nguyen et al. demonstrate that techniques like Projected Gradient Ascent (PGA) can accelerate the unlearning process by over 50x compared to retraining.
Another way to destroy specific feature representations is to train the model on the target data using intentionally corrupted information. Shakeri et al. propose a "learning-with-random-labels" objective. By locally retraining a task-specific module or prompt generator with randomized labels, the target client intentionally perturbs its own local weights, erasing learned patterns before the next global synchronization step occurs.
Standard Learning: Minimize Loss(Data, Correct Labels) ──> Sharp Features
Gradient Ascent: Maximize Loss(Data, Correct Labels) ──> Erased Features
Random Labels: Minimize Loss(Data, Random Labels) ──> Scrambled Features
To prevent this destructive process from ruining the model's performance on non-target data, Mora et al. introduce an on-device approach using knowledge distillation. The client device utilizes a "quasi-competent virtual teacher" that selectively penalizes the data meant to be forgotten while acting as a guardian to preserve general, unaffected knowledge directly on the device.
Mechanism 3: Collaborative Repair and Conflict Mitigation
Forcing a model to forget data—whether by subtracting historical gradients or maximizing loss—inevitably leaves a "hole" in the model's parameter space. Because the target client's data was entangled with peer models, removing it can degrade the accuracy of the remaining network or leave residual "leaked" knowledge in peer models (Ai et al.).
Therefore, effective federated unlearning requires a multi-stage procedure that pairs erasure with a collaborative retraining or repair protocol.
Step 1: Target Erasure ──> Erases client data but creates a parameter "hole"
Step 2: Collaborative Repair ──> Remaining clients fine-tune to restore global utility
As outlined by Liu et al., once the target client's influence is purged, both the target client (using neutral data) and unaffected clients participate in corrective updates. This ensures that the global model patches its performance gaps. Chen et al. validate this multi-stage strategy in sensitive domains like medical imaging, showing that a brief post-training phase on the remaining data successfully restores diagnostic accuracy across the rest of the federated network.
However, introducing unlearning updates can sometimes cause stability issues if they run counter to what the remaining clients need to know. To prevent this, Pan et al. introduce conflict mitigation strategies, such as projecting unlearning updates onto an orthogonal subspace. This ensures that the mathematical steps taken to erase the forgotten data do not conflict with or degrade the valid knowledge hosted by the remaining clients.
Key Foundational Work
Federated Unlearning for On-Device Recommendation (2022) — contributed 2 concept elements to this direction.
Empirical Study of Federated Unlearning: Efficiency and Effectiveness (2023) — contributed 2 concept elements to this direction.
Guest Editorial: Federated Optimizations and Networked Edge Intelligence (2022) — contributed 1 concept elements to this direction.