Bayesian Machine Unlearning via Posterior Approximation

This direction is merged into another direction: catastrophic forgetting in unlearning. Its last version stays readable and citable here.

Version 1 of 116 papers25 conceptsfrozen 2026-10-04

Machine unlearning is the process of removing the influence of specific training data from a model to satisfy privacy requirements like the "right to be forgotten." While standard machine learning models focus on point estimates of parameters, Bayesian models represent knowledge as a probability distribution—the posterior.

In this context, unlearning is not just about changing a weight; it is about transforming the entire posterior distribution to match what would have been learned if the deleted data had never existed.

The Bayesian Unlearning Objective

The gold standard for unlearning is "retraining from scratch" on the remaining dataset $D_{rem}$. If the original dataset was $D = \{D_{rem} \cup D_{f}\}$, where $D_{f}$ is the data to be forgotten, the goal is to recover the target posterior $p(\theta | D_{rem})$.

Retraining is computationally expensive. Bayesian unlearning researchers, such as Shaopeng Fu and Dacheng Tao, frame this as an approximation problem: how can we transform the current posterior $p(\theta | D)$ into the target posterior $p(\theta | D_{rem})$ without seeing all the data again?

Shifting Samples in MCMC

In sampling-based Bayesian methods like Markov Chain Monte Carlo (MCMC), the model's knowledge is stored as a "cloud" of samples. To unlearn data, one could theoretically re-run the entire Markov chain, but this is slow.

In Knowledge Removal in Sampling-based Bayesian Inference, Shaopeng Fu and colleagues propose a more efficient path: shifting the existing samples. By calculating an "MCMC influence function," they determine how each sample in the cloud would likely move if the forgotten data were removed. This "drifting" process nudges the existing sample distribution toward the target posterior, minimizing the KL-divergence between the current and the ideal state without requiring new samples to be generated.

Variational Inference and the Evidence Upper Bound

For models using Variational Inference (VI), knowledge is represented by a parameterized distribution $q(\theta)$. Unlearning here becomes an optimization task.

Bryan Kian Hsiang Low and Quoc Phong Nguyen introduced the Evidence Upper Bound (EUBO) in Variational Bayesian Unlearning. While standard VI maximizes the Evidence Lower Bound (ELBO) to fit data, unlearning minimizes the EUBO to "undo" the fit. This framework ensures that the model doesn't just forget the target data but also remains stable, preventing "catastrophic unlearning" where the model loses its general utility.

Similarly, in the context of generative models, Subhodip Panda and Prathosh AP apply this to diffusion models. In Variational Diffusion Unlearning, they derive a loss function that aligns the diffusion model's score function with a version of the model that never encountered the excluded classes or features.

Preserving Knowledge with Fisher Information

A recurring challenge in unlearning is ensuring the model doesn't "over-forget" and lose performance on the remaining data. Several researchers use the Fisher Information Matrix (FIM) as a regularizer to solve this.

The FIM represents how much information the training data carries about the model parameters. In RRL: Recommendation Reverse Learning, Xiaoyu You and Min Yang use FIM to identify which parameter relationships are critical for recommendation accuracy. By penalizing changes to these high-information parameters, they can surgically remove specific user interactions while keeping the rest of the recommendation engine intact.

This same principle is used by Prathosh AP in Partially Blinded Unlearning, where the pre-trained model acts as a prior, and the FIM ensures the updated weights stay close to that prior for all non-target classes.

Unlearning as Continual Learning

An intuitive way to view this field is as "reverse learning" or a specific form of Bayesian continual learning. Ryutaro Tanno and Yingzhen Li argue in Repairing Neural Networks by Leaving the Right Past Behind that model repair and unlearning are two sides of the same coin.

In their framework, identifying a "culprit" data point that causes a model failure is a diagnostic task, while removing its influence is a corrective Bayesian update. By treating the deletion of harmful data as a sequential learning step, they provide a unified way to update the posterior distribution to neutralize detrimental inputs while preserving existing knowledge.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_bayesian_machine_unlearning_via_posterio_bdab8e4e,
  title = {Bayesian Machine Unlearning via Posterior Approximation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bayesian-machine-unlearning-via-posterior-approximation/rdr_bdab8e4e73574aa7ad8db7d6925b75d3},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_bayesian_machine_unlearning_via_posterio_bdab8e4e_v1,
  title = {Bayesian Machine Unlearning via Posterior Approximation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bayesian-machine-unlearning-via-posterior-approximation/rdr_bdab8e4e73574aa7ad8db7d6925b75d3/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}