Adversarial Machine Unlearning

338 concepts192 papers1 proposal

Machine unlearning was originally conceived as a mechanism to enforce data privacy, allowing machine learning models to comply with legal frameworks like GDPR's "right to be forgotten" by erasing the influence of specific training samples. However, a deeper look at the mechanics of unlearning reveals that modifying a model's weights to delete a memory shares a profound mathematical boundary with adversarial security.

When we force a model to forget, we are actively reshaping its decision boundaries. This process can be viewed through two distinct lenses: as a powerful defense mechanism to surgically excise malicious exploits (such as backdoors and data poisoning) or as a dangerous attack surface that adversaries can weaponize to sabotage a model from within.


Unlearning as a Defense: Excising Malicious Memories

A backdoor attack injects a hidden trigger into a model's training data, forcing it to output a target misclassification whenever the trigger is present. From an optimization perspective, this backdoor is simply an unwanted feature-label association embedded deep within the network's parameters. Rather than retraining a compromised model from scratch—which is computationally expensive and can inadvertently re-incorporate residual poison—practitioners can treat the malicious trigger data as a "forget set."

Trigger Reverse-Engineering and Re-alignment

If a defender can approximate or reverse-engineer the visual or textual pattern of the hidden trigger, they can use targeted unlearning to break the malicious association.

Standard Backdoor:      [Clean Image + Trigger]  ───>  [Malicious Target Class]
Unlearning Defense:     [Clean Image + Trigger]  ───>  [Least Likely Class / Correct Class]

William Aiken et al. demonstrate that reintroducing reconstructed trigger patterns during a fine-tuning phase, but intentionally mislabeling them as the model's least likely class, forces the network to rapidly drop its confidence in the backdoor target. Alternatively, Shichang Sun et al. overlay synthetic versions of identified triggers onto clean images and retrain the model to map them back to their original, correct labels. As noted by Kealan Dunnett et al., this penalizes the shortcut associations formed during the poisoning phase, allowing the model to achieve true functional recovery rather than merely suppressing predictions.

Min-Max Optimization for Unknown Triggers

In realistic scenarios, defenders rarely know what the backdoor trigger looks like. To circumvent this, researchers frame backdoor removal as a minimax optimization problem that treats adversarial perturbations as proxies for the hidden triggers.

In this formulation, an inner loop generates a universal adversarial perturbation designed to maximize the model's classification loss. An outer loop then updates the model's weights or structural masks to minimize sensitivity to these worst-case perturbations.

  min          max      Loss( Model(X + Perturbation), Y_correct )
Model Weights  Perturbation

By solving this optimization challenge, Yi Zeng et al. (with their I-BAU framework) and Shuwen Chai et al. (via Adversarial Weight Masking) show that models can suppress latent vulnerabilities without requiring access to the original poisoned training data. The adversarial perturbations effectively find the "weak points" or shortcuts in the weight space, allowing the outer loop to smooth them out.

Feature Space Manifold Alignment

Instead of manipulating output labels, unlearning can operate directly on internal representations. Seonguk Seo et al. treat poisoned data as a forget set and align its feature representations with the eigenspace manifold of a clean "retain set." By forcing the model to encode poisoned inputs identically to clean inputs, the trigger loses its capacity to activate downstream malicious behavior.

Similarly, Yige Li et al. use attention distillation, where a clean-trained teacher network guides the attention maps of a backdoored student network, forcing it to ignore trigger-happy neurons and focus on legitimate task features.


Unlearning as an Attack Vector: Weaponizing Deletion

While unlearning offers an elegant way to sanitize models, the ability to dictate what a model forgets introduces a severe security vulnerability. If a system blindly executes user-submitted deletion requests, an adversary can manipulate the unlearning protocol to execute highly sophisticated attacks.

Inverse Data Poisoning

Traditional data poisoning requires an attacker to inject carefully crafted malicious samples into the training pipeline. "Inverse data poisoning," conceptualized by Chenxu Zhao et al., flips this mechanism. Instead of adding harmful data, the adversary strategically requests the removal of critical, benign training samples that maintain the model's decision boundaries.

Initial State:          [Class A Data]   <── Decision Boundary ──>   [Class B Data]
After Deletion Request: [Class A Data]   <── Boundary Shifts ────>   [Critical Data Removed]

By stripping away the foundational support vectors or boundary-defining samples of a specific class, the adversary causes the model's decision boundaries to collapse or shift dramatically. This allows them to steer the model toward a target malicious state or introduce severe performance degradation and fairness biases under the guise of a legitimate privacy request.

Camouflaged Poisoning (The Sleeper Agent)

Unlearning can also act as a temporal trigger or a delayed fuse for a backdoor. Jimmy Z. Di et al. expose a vulnerability where an attacker injects both poisoning data and "camouflage data" into a model during initial training. The camouflage data is specifically designed to neutralize or mask the performance impact of the poison, keeping the backdoor dormant and undetectable by standard validation checks.

Phase 1 (Training):   [Poison Data] + [Camouflage Data]  ───>  Backdoor is Dormant
Phase 2 (Unlearning): User requests removal of Camouflage ───>  Backdoor Activates

When the attacker later submits a routine request to remove the camouflage data, the victim's unlearning algorithm updates the model's parameters. Erasing the camouflage unmasks the underlying poison, instantly activating the sleeper-cell backdoor in the deployed model.

Over-Unlearning and Trajectory Sabotage

The structural fragility of approximate unlearning algorithms makes them vulnerable to exploitation. Shuo Wang demonstrates the threat of "over-unlearning," where an adversary crafts a malicious unlearning request that causes catastrophic collateral damage. Instead of forgetting a single user's data, the algorithm experiences a disproportionate utility loss across entire unrequested classes, effectively sabotaging Machine Learning as a Service (MLaaS) platforms through simple black-box API interactions.

Furthermore, Vinith Menon Suriyakumar et al. prove that online unlearning algorithms—which approximate data removal via techniques like the Infinitesimal Jackknife—are fundamentally vulnerable to trajectory manipulation. Because these algorithms rely on smooth optimization paths, an attacker can introduce data designed to warp the model's hyperparameter landscape upon deletion, ensuring that the post-unlearning model state lands in a highly degraded or compromised region of the parameter space.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 194 papers
  2. Version 1 · 2026-10-04 · 172 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_adversarial_unlearning_4e6dd2c6,
  title = {adversarial unlearning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/adversarial-unlearning/rdr_4e6dd2c6227a4b31baab472be47924a7},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_adversarial_unlearning_4e6dd2c6_v2,
  title = {adversarial unlearning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/adversarial-unlearning/rdr_4e6dd2c6227a4b31baab472be47924a7/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake