Efficient Data Removal in Federated Optimization

This direction is merged into another direction: right to be forgotten. Its last version stays readable and citable here.

Version 1 of 119 papers21 conceptsfrozen 2026-10-04

When a user exercises their "right to be forgotten" under regulations like GDPR, a machine learning model must erase the influence of that user's data. In a standard centralized setting, this is computationally expensive; in Federated Learning (FL), where data is distributed across thousands of devices, it is a logistical nightmare.

The naive solution is to retrain the model from scratch without the target data. However, for large-scale models, this is prohibitively slow and energy-intensive. Modern research focuses on Federated Unlearning (FU): techniques that "subtract" a client's contribution from a trained model while maintaining high accuracy and provable privacy guarantees.

The Storage-Time Trade-off

The fundamental challenge in unlearning is the storage bottleneck. To undo a training step, you theoretically need to know what the model looked like before that step occurred.

Early frameworks like FedEraser, introduced by Gaoyang Liu, prioritize reducing unlearning latency by storing intermediate model weights. By reconstructing the training history from these checkpoints, the system can "roll back" and recalculate a clean training path. However, as Rashmi Gangadharaiah and Swanand Kadhe point out, storing full-model checkpoints for massive language models (LLMs) is often unsustainable, requiring massive disk space that scales poorly with the number of training rounds.

Informed Restarts and Sensitivity Tracking

To make roll-backs more efficient, researchers have moved toward "informed" strategies. Instead of rolling back to the very beginning, the goal is to find the latest possible checkpoint that hasn't been "too corrupted" by the data to be forgotten.

Yann Fraboni and colleagues developed SIFU (Sequential Informed Federated Unlearning). This framework tracks the "sensitivity" of the global model to each client during the initial training. When an unlearning request arrives, the server identifies an optimal historical checkpoint, applies a calculated burst of noise to wipe the specific client's influence, and performs a brief retraining phase. This approach, also explored in Informed Federated Unlearning, provides mathematical guarantees that the resulting model is statistically indistinguishable from one that never saw the deleted data.

Unlearning via Gradient and Weight Manipulation

A more radical approach avoids checkpoints entirely by treating unlearning as an optimization problem in reverse. If training is a series of "nudges" (gradients) toward a minimum, unlearning can be viewed as a "counter-nudge."

  • Pseudo-Gradient Negation: Alessio Mora introduced PUF, which simply negates and scales the historical updates from the client requesting to be forgotten. This requires zero additional storage.
  • Weight Negation: Yasser H. Khalil proposed NoT, which disrupts the "co-adaptation" of model layers by multiplying specific weights by -1, followed by a quick fine-tuning to restore performance.
  • Subspace Projection: To ensure that "erasing" one client doesn't destroy the knowledge gained from others, Guanghao Li developed SFU. This method projects the unlearning gradient into a subspace orthogonal to the remaining clients' data, effectively surgical-striking the target data's influence without collateral damage to model accuracy.

Structural Isolation and Linearization

Some researchers argue that models should be designed for unlearning from the start. By changing the architecture or the training objective, the "influence" of data can be localized.

In the context of Knowledge Distillation, Yijun Quan proposed PURGE. By partitioning how different "teacher" models influence a "student" network, the system can unlearn data by only retraining a small subset of the student's parameters.

For deep neural networks, Ruinan Jin introduced F²L², which linearizes the model. This transformation allows the server to perform a "certified" Newton step—a single mathematical operation that removes data traces (like malicious backdoors) without needing to communicate with clients or store historical models.

Beyond Accuracy: Fairness and Clinical Reality

As these techniques mature, the focus is shifting toward the side effects of unlearning. Swanand Kadhe warns that the unlearning process can inadvertently introduce social biases, necessitating post-processing frameworks like FairSISA to balance accuracy and equity.

Furthermore, in sensitive fields like medical imaging, Juntao Hu emphasizes that unlearning is not just a technical hurdle but a clinical necessity for maintaining long-term regulatory compliance in heterogeneous, privacy-first environments.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_efficient_data_removal_in_federated_opti_c03c6a73,
  title = {Efficient Data Removal in Federated Optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/efficient-data-removal-in-federated-optimization/rdr_c03c6a7300dc47c88c30c4f81afbfdd2},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_efficient_data_removal_in_federated_opti_c03c6a73_v1,
  title = {Efficient Data Removal in Federated Optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/efficient-data-removal-in-federated-optimization/rdr_c03c6a7300dc47c88c30c4f81afbfdd2/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}