When a trained model must forget some of its training data, the usual approach nudges its weights away from that data by gradient ascent or a bounded variant of it. Because the same weights also carry everything else the model knows, these nudges often damage the rest of the model. This note proposes a different operation. First, represent a trained network's knowledge as a small summary stored at a few hundred "inducing points": virtual anchor inputs through which, in a sparse Gaussian process (a probabilistic model over functions that normally costs cubic time in the dataset size), all data points communicate. Once the knowledge is in that form, unlearning becomes subtraction. Each training point contributes a positive term to the summary. Removing the forget set means computing its terms and subtracting them, which leaves exactly the summary the retained data would have produced. There is no search, no unbounded loss, and no division of probability densities.
The case for doing this now comes from a specific failure in the literature. Challenges and Pitfalls of Bayesian Unlearning showed that "Bayes' rule in reverse" in weight space, whether by Laplace or variational approximation, is brittle. It swings between not unlearning and collapsing, precision matrices lose positive semi-definiteness, and it gives overconfident predictions exactly where data was erased. That paper explicitly left open whether function-space or inducing-point approximations avoid these pathologies. We argue that they do, identify the precise reason, and point out a leak the obvious version misses: the inducing point locations themselves can reveal the forgotten data.
Core Claim
We start from the Sparse Function-space Representation (SFR) of Function-space Parameterization of Neural Networks for Sequential Learning. SFR linearizes a trained network around its weights $\theta^*$ and turns it into a Gaussian process whose kernel is built from network Jacobians. Each training point $i$ then gets two scalars, called dual parameters: $\alpha_i$, which drives the mean prediction, and $\beta_i \ge 0$, which drives the uncertainty. These are projected onto $M$ inducing inputs $Z$. The entire posterior is fixed by two aggregates,
$$ A = \sum_{i} \beta_i\, k_Z(x_i)\, k_Z(x_i)^\top, \qquad b = \sum_{i} \alpha_i\, k_Z(x_i), $$
where $k_Z(x_i)\in\mathbb{R}^M$ is the kernel between input $x_i$ and the inducing points. The posterior precision over the function values at $Z$ is $K_{ZZ}^{-1} + K_{ZZ}^{-1} A K_{ZZ}^{-1}$, with $K_{ZZ}$ the kernel matrix among inducing points. SFR used these sums only to add new data.
Our proposal runs them in reverse. We evaluate the duals of the forget set at $\theta^*$ and set $A_r = A - A_f$ and $b_r = b - b_f$. This operation has two properties. First, $A_r$ is by construction a sum of positive semi-definite terms from retained points, so the result is always a valid Gaussian. It cannot become non-normalizable, because we never divide densities; we only remove terms whose sum we can recompute. Second, wherever the forget set was the only data, predictive variance returns to the prior. That is the restored ignorance the Pitfalls paper found missing.
The exactness holds relative to a fixed linearization point and a fixed $Z$. Both were chosen using the forget data, and that is where the real research lies.
Our diagnosis of the prior failure makes this sharper. Weight-space Laplace downdating (L-BUN) has the same additive structure in principle. However, with millions of parameters it must approximate the precision with diagonal, Kronecker-factored, or empirical-Fisher forms. Subtracting a forget term computed in a different approximation from the one that was accumulated breaks the sum structure and destroys positive semi-definiteness. The inducing-point summary is small enough, $M\times M$, to store and downdate exactly. It is still expressive enough to act as a surrogate for the network.
Research Program
The first component is the downdate itself. We compute forget-set Jacobians and duals at $\theta^*$, form $A_f$ and $b_f$, and apply a Cholesky downdate. The cost is $O(|D_f| M^2 + M^3)$, with no backpropagation through the network. We will prove the positive semi-definiteness and prior-reversion properties under log-concave likelihoods. We will also measure numerical cancellation when the forget set dominates $A$.
The second component addresses inducing-point leakage and relocation. Variational Bayesian Unlearning keeps $Z$ fixed, but inducing points selected or optimized on the full data sit near forget points. An attacker who can read the model therefore learns where the deleted data was, which is a structural Streisand effect of the kind Deep Regression Unlearning found at the output level. We will prune inducing points whose nearest-neighbour mass is dominated by the forget set. We will reseat them by greedy variance-based selection from retained inputs, and then re-project the retained statistics onto the new $Z$. The re-projection needs one forward-only pass over retained data. That costs more than the bare downdate but far less than retraining, and we will report both regimes.
The third component corrects for the linearization point. $\theta^*$ still encodes the forget data, and the Jacobian kernel inherits it. This is the function-space form of the "illusion of unlearning" in An Illusion of Unlearning? Assessing Machine Unlearning Through Internal Representations. We will add an optional re-linearization step. It runs a few epochs of retain-only fine-tuning, regularized so that the network's function values at $Z$ match the downdated posterior mean, using SFR's function-space regularizer. Afterwards we recompute the summary. The question is how far the one-shot downdate gets on its own, and when re-linearization is necessary. The Taylor expansion should fail when the forget set shaped the learned features themselves, as in whole-class removal, and should hold for scattered or regional removals.
Baselines
The naive reference is the original model with no unlearning. The gold reference is full retraining on the retained data. Every metric is reported as a distance to the retrained model, which bounds the comparison from both sides.
The closest existing method is EUBO minimization from Variational Bayesian Unlearning. It optimizes the inducing-point variational distribution to undo the forget likelihood, keeping $Z$ fixed. We run it on the same sparse GP, with the same $Z$, $M$ and kernel, and give it a hyperparameter sweep of the same wall-clock budget as our full pipeline. On Airline it removed 100,000 of 2 million points "significantly faster than full retraining." No reported accuracy or NLPD value for that result was found on Lacuna, so we will reproduce it. Beating it means matching or lowering its NLPD gap to retraining without its tail-driven instabilities, which we will count as the fraction of sweep settings that diverge.
The weight-space Bayesian baseline is L-BUN from Challenges and Pitfalls of Bayesian Unlearning, run on the same small network in two forms. One uses the exact full precision, which is feasible only at small scale. The other uses a Kronecker-factored precision. This pair is the ablation that isolates our key ingredient. If exact weight-space downdating matches us while Kronecker downdating fails, the benefit comes from exact additive statistics at a tractable size, not from function space as such. That is the honest version of the claim.
The neural unlearning baselines are negative gradient and Blindspot from Deep Regression Unlearning for regression. For classification they are Negative Preference Optimization from Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning and selective removal from Distributional Machine Unlearning via Selective Data Removal, using its released code. All share the same trained network and a fixed GPU-hour budget. On AgeDB, Blindspot took 72 seconds against 1,666 for retraining. On CIFAR-10, selective removal kept retained-class accuracy within 0.03 absolute variation. These are the numbers to meet on speed and retain utility while also improving uncertainty in the forget region. Internal ablations toggle relocation of $Z$ and re-linearization independently.
Why not just…?
Why not just run Negative Preference Optimization, which already avoids collapse? NPO bounds the forget loss so that collapse is slow. It still moves shared weights, however, and it offers no notion of restored uncertainty: an unlearned regression model under NPO has no variance to return to the prior. This is a real threat on the classification tasks where it applies. The CIFAR-10 class-removal comparison will settle whether our downdate plus re-linearization matches its retain accuracy. If it does not, the contribution narrows to calibrated regional unlearning.
Why not just do exact Laplace downdating on the last layer? For a last-layer model with a small feature dimension, this is equivalent in the linear-Gaussian case, and we expect it to work. The exact-versus-Kronecker L-BUN ablation tests exactly this. Our method matters when the Jacobian spans more parameters than any exact weight-space precision can hold. We will report the parameter count at which exact weight-space downdating stops being feasible and what the last-layer version loses against full-Jacobian SFR in NLPD gap to retraining.
Why not just fine-tune on the retained data, as Blindspot and gradient-difference methods do? Retain-only fine-tuning is essentially our re-linearization step without the downdated target. The ablation that runs it with and without the function-space anchor at $Z$ shows whether the closed-form downdate supplies the useful signal, or whether fine-tuning alone gets there with equal compute.
Proposed Experiments
The first decisive check uses the 1D sine benchmark from the Pitfalls paper: 300 points with the central 200 deleted. Success means the predictive variance in the gap matches retraining, and the result is insensitive to hyperparameters. It has no unlearning hyperparameter except the choice of $M$. The paper reports this failure qualitatively; no reported numerical result was found on Lacuna.
At scale, we remove geographic and temporal slices from Airline and UK Housing, the latter with an SVIGP mean squared error of 0.426 from Gaussian Processes for Big Data, using the sparse GP directly. We then apply the SFR pipeline to deep regression networks on AgeDB and UCI Electricity. Metrics are the Wasserstein distance to retrained outputs, retain and forget error, NLPD in the forget region, and membership inference success.
A new attack reads only $Z$ and tries to infer forget-set membership. We run it with and without relocation to test the leakage hypothesis directly.
On CIFAR-10 single-class removal and on Split-MNIST task removal, we test the regime where linearization should break. We report how much of the gap to retraining re-linearization closes.
The note is falsified in two cases. The first is if the one-shot downdate fails to beat EUBO on the regional-removal tasks. The second is if exact weight-space downdating matches it at every scale we can run. In that case the right conclusion is that the Pitfalls failures came from approximated precisions, not from weight space.
References
- Challenges and Pitfalls of Bayesian Unlearning
- Function-space Parameterization of Neural Networks for Sequential Learning
- Variational Bayesian Unlearning
- Deep Regression Unlearning
- An Illusion of Unlearning? Assessing Machine Unlearning Through Internal Representations
- Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning
- Distributional Machine Unlearning via Selective Data Removal
- Gaussian Processes for Big Data