Forgetting as Disagreement: Retain-Free Unlearning Inside Low-Cost Ensembles

When a model must forget some of its training data, the usual fix fine-tunes it on the data to forget and also on the data to keep. In practice the data to keep has often been deleted or locked away. Methods that work without it are called zero-shot or retain-free unlearning. These methods usually aim for the wrong end state. They push the model to be wrong or maximally unsure about the forgotten examples. But a model that never saw those examples would not behave that way. It would treat them as unfamiliar inputs, and an ensemble of such models (several networks whose predictions are averaged) would disagree about them. Ensembles are the standard way to measure this kind of "I have not seen this" uncertainty, called epistemic uncertainty. Their weakness is cost: $M$ networks mean roughly $M$ times the memory and compute.

This note proposes to unlearn without retained data by editing only the small member-specific parameters of a low-cost ensemble. In BatchEnsemble-style models, all members share one set of weights and each member adds its own rank-one scaling vectors. The edit drives members to disagree on the forget set, by the same amount that they already disagree on genuinely unseen inputs. Two things fall out of this design. The shared weights stay frozen and act as an implicit stand-in for the missing retain set. The model's own disagreement on unseen data gives a label-free target for tuning hyperparameters, which is the main practical failure of current zero-shot methods. The deployed model keeps exactly its original inference cost, and the edit is a few kilobytes per member.

Core Claim

In a low-cost ensemble, retain-free unlearning should be posed as matching the epistemic uncertainty a retrained ensemble would show on the forget set. It should not be posed as minimizing accuracy or maximizing output entropy on that set. Restricting the edit to the per-member factors does two jobs. It preserves retained behavior without any retained data. It also lets the edited model reach that disagreement target at unchanged deployment cost.

The measured quantity is the members' mutual information at input $x$, which is a standard way to measure how much the members disagree:

$$I(x) = H\Big(\tfrac{1}{M}\textstyle\sum_m p_m(y\mid x)\Big) - \tfrac{1}{M}\textstyle\sum_m H\big(p_m(y\mid x)\big)$$

Here $p_m$ is member $m$'s predictive distribution and $H$ is entropy. If every member outputs the same uniform distribution, $I(x)=0$. That is the trap that output-entropy methods fall into. The unlearning objective updates only the rank-one factors $\{r_m, s_m\}$, with the shared weights frozen. It minimizes

$$\mathbb{E}_{x\in D_f,\ \epsilon}\big[\,(I(x+\epsilon) - \tau)^2 + \lambda\, \bar p(y_x \mid x+\epsilon)\big] + \mu \sum_m \|(r_m,s_m)-(r_m^0,s_m^0)\|^2 .$$

The symbols are as follows. $D_f$ is the forget set, and $\epsilon$ is small Gaussian noise; this borrows the neighborhood idea of JiT so that the edit covers a region around each point rather than the point alone. $\bar p(y_x\mid\cdot)$ is the ensemble-mean probability of the original label. The superscript $0$ marks the factors before unlearning. The target $\tau$ is the median disagreement the original ensemble shows on a reference pool of never-seen images, for example COCO crops as used by SCAR. No retained data enters anywhere.

Relation to Prior Work

Three retain-free methods supply mechanisms that this proposal reuses or argues against.

Zero-Shot Machine Unlearning at Scale via Lipschitz Regularization (JiT) smooths a single model's outputs over noisy neighborhoods of forget points. It reports being "over 5x faster than" the prior zero-shot method Boundary Shrinking. It also notes that its noise scale and learning rate are hard to tune without retained data. We reuse its neighborhood sampling, and our disagreement-matching target is aimed directly at that tuning problem.

Is Retain Set All You Need in Machine Unlearning? (SCAR) replaces the retain set with out-of-distribution images used as distillation anchors. It reports an Adaptive Unlearning Score of about 0.93–0.98 on CIFAR-10, CIFAR-100 and TinyImageNet. We borrow its anchor pool, but only to set $\tau$, not to distill.

Partially Blinded Unlearning (PBU) uses a Fisher-weighted stability term computed only on the forget set. Like SCAR, it produces a single point predictor with no uncertainty model.

Two papers motivate the choice of target. Evaluating Machine Unlearning via Epistemic Uncertainty shows that a model can have 0% accuracy on a forgotten class while keeping low epistemic uncertainty. It also warns that unlearning can make forgotten points stand out as anomalous, which it calls a "Streisand effect." That paper uses uncertainty only to evaluate unlearning; we optimize it. Unleashing Uncertainty (SAFEMax) maximizes output entropy in diffusion models. Uniform agreement among members is exactly the signature our mutual-information term is built to avoid, so SAFEMax-style entropy maximization becomes one of our ablations.

The ensemble side supplies the deployment setting. Improving Ensemble Distillation With Weight Averaging and Diversifying Perturbation (LatentBE) trains rank-one BatchEnsemble members in a shared low-loss region and then averages them. It reports 0.107 ms/img against 0.390 ms/img for a 4-model ensemble on ImageNet, and a CIFAR-100 NLL of 0.792 against 1.038 for standard distillation. Averaging into one network would erase the member disagreement we rely on. We therefore deploy the unaveraged rank-one ensemble, whose memory is close to that of a single network, and study the averaged variant as a separate case.

Two papers constrain the design. Efficient Verified Unlearning For Distillation (PURGE) is the only work in the file that combines ensembles with unlearning. It needs the partitioned retained shards and leaves inference cost at full multi-model scale. Our setting is defined by the absence of those shards. Distillation Robustifies Unlearning (UNDO) shows that behavioral unlearning leaves latent circuitry that a few fine-tuning steps can reactivate. This is the most serious threat to an edit confined to rank-one factors, so relearning attacks are a primary evaluation rather than an afterthought.

Research Program

The first stage builds the testbed. We train rank-one BatchEnsembles ($M=4$, matching LatentBE) with ResNet backbones on CIFAR-10, CIFAR-100 and TinyImageNet. For every unlearning request we also retrain a "gold" ensemble on $D_r$. Each gold ensemble gives the reference distribution of mutual information on $D_f$ that a perfect unlearner should reproduce. Requests come in two kinds: full-class forgetting, and random-sample forgetting (1% and 10% of the training set). The sample case matters because the forget points there are in-distribution, so the right target disagreement is lower than in the class case.

The second stage studies whether $\tau$, taken from unseen reference images, approximates the gold ensemble's disagreement on $D_f$. We will check the class case and the sample case separately. If it holds only for classes, we will instead calibrate $\tau$ from the original ensemble's disagreement on held-out test data. Test data counts as unseen data, not retained data, and is often available even when the training set is not.

The third stage is the edit itself. We compare factor-only edits against shared-weight edits and against single-layer shared edits in the style of SLUG (Targeted Unlearning with Single Layer Unlearning Gradient). We expect factor-only edits to leave little to undo but possibly too little capacity to remove features. The fourth stage stress-tests the edited models. We run relearning attacks following UNDO, membership inference, and the representation clustering attack of DECAF. The fifth stage scales to a ViT on ImageNet subsets, as JiT did.

Baselines

The naive reference is gradient ascent or random relabeling on $D_f$, applied to the shared weights of the same ensemble. JiT and SCAR both report that this collapses utility, and it fixes the floor.

The strongest existing competitor is JiT applied independently to each member of a full deep ensemble of four separately trained networks. This is the most faithful version of "use a known zero-shot method and keep ensemble uncertainty." It costs four times as much at unlearning and at inference. A second strong competitor is SCAR applied per member, with the same COCO anchor pool. For both, there is no reported number found on Lacuna for JiT accuracy on CIFAR, so we will reproduce it. For SCAR, the reference is its reported Adaptive Unlearning Score of about 0.93–0.98 across CIFAR-10, CIFAR-100 and TinyImageNet.

PBU applied to the rank-one ensemble is a further baseline. Its reported claim is 5–20% higher retained accuracy than Bad Teaching and Fast-Effective on CIFAR-100 and Food-101.

Fairness rules apply to every competitor. Each sees the same forget sets, the same reference pool, and no retained data. Each gets the same hyperparameter-search budget of 20 configurations. Configurations are selected by each method's own label-free criterion, never by peeking at $D_r$. We also report an oracle selection using $D_r$ for every method, which separates algorithm quality from validation quality.

Two ablations isolate the proposal's key ingredients. The first edits the shared weights with our objective. The second replaces the mutual-information term with ensemble-mean entropy maximization, which is the SAFEMax-style target. A third ablation gives each member a different random wrong label, a crude way to manufacture disagreement.

Beating the per-member JiT/SCAR baseline means three things together. Our disagreement distribution on $D_f$ should be closer to the gold ensemble's, measured by Wasserstein distance. Retained accuracy and NLL should be within one point of that baseline. And this should hold with a single edited model at one quarter the deployment memory.

Why not just…?

Why not just unlearn a single network with JiT or SCAR and skip ensembles entirely? That answers a different question. The systems at issue are deployed with uncertainty estimates, because they drive OOD rejection and selective prediction. A single point predictor cannot show whether forgotten points now look unfamiliar. The epistemic-uncertainty evaluation paper shows that low accuracy alone hides retained information. We will still report single-model JiT and SCAR on accuracy-based metrics, so readers can see whether anything is lost on those metrics.

Why not just apply JiT to every member of a full deep ensemble? This is a real threat on forgetting quality. Independently edited members may well disagree on $D_f$ without any explicit target. The proposal does not claim to beat it on quality at any cost. It claims to match it at four times lower deployment memory and unlearning compute, with a principled way to tune hyperparameters. If per-member JiT is clearly closer to the gold disagreement, the factor-only design loses. Our headline comparison is set up to show exactly that if it happens.

Why not just maximize output entropy on the forget set, as SAFEMax does? Members that all agree on a uniform distribution produce maximal entropy but zero disagreement. That is precisely the anomalous signature that an uncertainty-aware attacker or the Streisand-effect critique would flag. The entropy ablation tests this directly. We predict it matches our method on forget accuracy but separates from the gold ensemble on mutual information and on membership inference against an attacker that uses member disagreement.

Why not just give members different random labels on $D_f$? It manufactures disagreement cheaply, so it is included as an ablation. Without the noisy neighborhoods and the $\tau$ target, we expect it to overshoot on sample-level forgetting, where gold disagreement is low. That overshoot would make forgotten points detectable.

Proposed Experiments

The first decisive check compares the reference-pool target with the gold-ensemble target. If $\tau$ from COCO or test data predicts gold disagreement on $D_f$ poorly for random-sample forgetting, the label-free tuning story is limited to class forgetting. We would report it that way.

The second check is relearning. Following UNDO, we fine-tune each unlearned model for up to 100 steps on a small labeled slice of $D_f$ and plot recovered forget accuracy against steps, with the gold ensemble as reference. If factor-only edits recover much faster than shared-weight edits, the frozen backbone still holds the forgotten features. In that case the method needs a follow-up step, such as noise added to the shared weights in the style of UNDO.

The third check is deployment cost. We report memory, MACs and latency before and after unlearning. We also plug the models into a window-based early-exit cascade following Window-Based Early-Exit Cascades for Uncertainty Estimation. The question is whether forget queries, now uncertain, get routed through every member, and whether that raises compute on those queries in a way an observer could detect.

The final check is OOD detection on the retained task. We use CIFAR-100 against SVHN and TinyImageNet, with SNGP's reported AUROC of 0.846 ± 0.019 on CIFAR-100 (A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awareness) as an external anchor. This confirms that the edit did not degrade the ensemble's uncertainty quality elsewhere.